As software moves to the daily routines and responsibilities of people, there is a need for developing tools and languages rapidly. Domain-Speci c Languages (DSLs) are claimed to contribute to this productivity increase, while reducing the required maintenance and programming expertise. DSLs are designed to bridge the gap between the problem domain (essential concepts, domain knowledge, techniques, and paradigms) and the solution domain (technical space, middleware, platforms and programming languages). The sooner we ll in this gap, the sooner we shall increase users productivity. However intuitive this idea may be, we need to have means to assert the quality and success of the developed languages. The alternative is to accept the risk of deriving inappropriate products that bring more harm by decreasing productivity or even increasing maintenance costs.
The software industry does not seem to report investment on the evaluation of
DSLs, as shown in a recent systematic literature review [
It is arguable that the main reason for the perceived high costs of DSL
evaluation, is the lack of a consistent and computer-aided integration of two di erent
and demanding complementary processes: SLE process, and language evaluation
process. On the one hand, language engineers should become aware of quality
concerns during language development, and identify and apply best practices
into their development plan. On the other hand, evaluation experts should get
better understanding of all the models involved in the software language
development in order to be able to give appropriate and reliable suggestions towards
the improvement of the DSL under development. The focus of this research is
to propose systematic evaluation process for DSLs with usability concern [
A programming language is a model that describes the allowed terms and how
to compose them into valid sentences. DSLs that are interest of this research are
generally conceived as communication interfaces between human and computers
[
Empirical, or experimental, evaluation studies of UI with real users is a
crucial phase of its validation [
There is an increasing awareness to the quality in use of languages, fostered by the competition of language providers. Better usability is a competitive advantage, although evaluating it remains challenging. While evaluating competing languages it is hard to interpret the existing metrics in a fair, unbiased way, provide reliable design changes and assure that scope of evaluation is preserved to target user groups.
When we consider General Purpose Languages (GPLs), their users are part of population that master well mathematical and technical concepts. In order to develop programming solutions they need to master also domain concepts. On other hand, as DSLs are meant to reduce use of computation domain concept by putting focus on the domain concepts, they are expected to be used by the much diverse target population.
The increased productivity achieved by using DSLs, when compared to using
GPLs, is one of the strongest claims by the DSL community. With anecdotal
boosted speed development reports of DSLs ([
As result of inspection of current methodologies and tools for evaluation of UIs
and GPLs we propose an iterative user-centered approach for evaluation of DSLs.
Goal of this approach is to establish formal correspondences between the DSL
development process and the experimental evaluation at all the stages [
Approach is described by set of patterns that are introduced in order to
provide a complete solution to a complex problem of placing intended users as a
focal point of DSLs design and conception, and by that ensure that the language
satis es the user requirements [
The patterns are divided in three spaces (see Fig.1), that represent di erent level of abstraction. Agile Development Process gives set of patterns devoted to project management and engineering of a DSL. This is the most important set of patterns, as it is trough organization and planning of language development and evaluation activities and goals we are controlling and tracking success of produced language. After an iteration, goals are scoped and budget is xed, we are ready to proceed to design and implementation activities that are guided by patterns given in Iterative User-Centered Design pattern space. As the users are the central part of a DSL evaluation, this patterns considers how to engage the user in the development process and how to collect valuable information about the DSL and its level of usability while it is being developed. Finally, they are expected to result with concrete hypothesis, tests, metrics, samples and statements that should be addressed and validated trough Experimental Evaluation Design.
Following these best practices, each development iteration is focusing on different increment or level of abstraction that will be evaluated or re ned. By planing carefully development process and organization of responsibilities and costs, goal is to establish balanced menagment and engineering plan that will satisfy both: business and user needs, by optimizing impact of evaluation feedback on language development.
According to this proposal, language and evaluation goals should be identied during domain engineering phase of DSL's construction i.e. while eliciting minimum set of domain concepts. A rst step would be to understand and specify the context of use of DSLs and which kind of user groups it should target by constructing User and Context model. In order to achieve that, interviews or questionnaires with the DSL's intended end users should be designed in order to capture information about their working environment and the baseline approach to solve problems. In the language design phase, it is necessary to identify which quality attributes are impacted by the implementation of which domain concepts or layer of abstraction. During the implementation phase, the language engineer can bene t from the collected information by means of tools or instruments that implements chosen measures directly on the DSL prototype. Finally, in the testing phase, the language engineer should conduct (at least) an expert evaluation to validate that the known quality problems and functional tests passed well. When seams that evaluation goals are met, we should conduct a user-based evaluation, in a real context of use, to assess the DSL's quality in use. That is done by giving the users real problems to solve in order to cover the most important tasks identi ed in the domain. Data about satisfaction and cognitive workload should also be evaluated subjectively through questionnaires. It is especially important in this phase to measure all the learnability issues, since DSLs should be (in principle) easy to learn and remember. 4
Under the perspective of SLE, in order to experimentally evaluate a DSL, we
need to know what is the criteria involved, understand notion of quality from
the relevant perspectives and understand the experimental process itself. This
complex challenge with respect to reuse was covered by general model for DSL
experimental evaluation presented in [
Experimental model outlines the activities needed to perform an
experimental evaluation of a software engineering claim, following the scienti c method.
In order to e ectively reason about experimental process and eventually
detect aws before it is applied and analysed we systematically compared four
language evaluation experiments ([
By allowing signi cant changes to correct de ciencies along the development
process instead of just evaluating at the end of it (when it might be too late),
presented user-centered design is ment to reduce development and support costs,
increase sales, and reduce sta cost for employers [
Acknowledgments I gratefully thank to my supervisors Vasco Amaral and Miguel Goul~ao . This work was partially supported by the CITI - PEst - OE /EEI /UI0527 /2011, (CITI/FCT/UNL) - 2011-2012)