To deal with the increasing complexity of the software systems to be developed, it is important to have high level approaches to development that support such complexity at a ordable costs. This fosters the development of Domain-Speci c Languages (DSLs) that are claimed to bring important productivity improvements to software developers. Because of that, the quality of the users interaction with this kind of technology is becoming of utmost importance. The goal of this research is to contribute to the systematic activity of software language engineering by focusing on the issue of quality in use evaluation of DSLs, in which design decisions are validated iteratively.
As software moves to the daily routines and responsibilities of people, there is a
need for developing tools rapidly. Domain-Speci c Languages (DSLs) are claimed
to contribute to a productivity increase in software systems development, while
reducing the required maintenance and required programming expertise. DSLs
are designed to bridge the gap between the problem domain (essential concepts,
domain knowledge, techniques, and paradigms) and the solution domain
(technical space, middleware, platforms and programming languages). DSLs empower
domain experts to specify systems for their own domains. However intuitive this
idea may be, we need to have means to assert the quality and success of the
developed languages. We argue that for this kind of software languages the
measure of success has to be captured by assessing the real impact of using DSL
in real context of use by their target domain users. The alternative is to accept
the risk of deriving inappropriate products that bring more harm by decreasing
productivity or even increasing maintenance costs at the long run.
Domain-driven development is becoming increasingly popular, as it raises the
abstraction level. Language engineers need to deal with the accidental complexity
of the used computer technology, e.g., the use of low level abstraction
programming languages, while integrating a wide plethora of di erent tools and libraries
to create a DSL for target domain context. On the other hand, DSL
development requires domain and language development expertise, and few people have
both. This lead language engineers to cope with the growing of both essential
and accidental complexity [
Software language engineering (SLE) is the application of a systematic,
disciplined and quanti able approach to the development, usage, and maintenance of
software languages [
The software industry does not seem to report investment on the evaluation
of DSLs, as shown in a recent systematic literature review [
The focus of this research is to propose a systematic evaluation process for
DSLs with the usability concern [
A programming language is a model that describes the allowed terms and how
to compose them into valid sentences. DSLs are generally conceived as
communication interfaces between human and computers. User Interfaces (UIs) can be
also seen as a realization of a language. From this perspective it was discussed in
[
Empirical (i.e. experimental) evaluation studies of UI with real users is a
crucial phase of UI validation [
There is an increasing awareness to the quality in use of languages, fostered by the competition of language providers. Better usability is a competitive advantage, although evaluating it remains challenging. While evaluating competing languages, it is hard to i) interpret the existing metrics in a fair, unbiased way, ii) provide reliable design changes; and iii) assure that the scope of evaluation is preserved to target user groups.
In the context of General Purpose Languages (GPLs), comparing the
productivity impact of di erent languages during development process has some
tradition [
A systematic approach based on UI experimental validation techniques should be used to assess the impact of new DSLs. For that purpose, it is necessary to merge common usability evaluation process with the DSL development process. We expect that introducing evaluation experts into DSL development will enable language engineers to build languages that are easier to use, leading to increased productivity and positive experience by their users.
In order to evaluate DSLs we need a rigorous collaborative procedure (both during and after their development), as well as evaluate their sentences (called instance models). For that it is necessary to (i) de ne the usability criteria to evaluate DSLs by focusing on indicators that are relevant for users perception of quality (e.g. e ciency, e ectiveness, reliability,learnability, scalability, reusability ect.); (ii) integrate in an existing IDE support Quality in Use indicators; and (iii) de ne a methodological approach to support the evolution of a DSLs based on experience and infer its impact on quality improvement during its lifecycle (e.g. traceability of design decisions, reuse of experiment results in a domain).
An iterative user-centered approach DSL for evaluation was proposed in [
These patterns outline di erent roles relevant for development and evaluation
of DSL, and are divided into three sets that correspond to a di erent abstraction
space: (i) Agile Development Process includes patterns devoted to project
management and engineering of a DSL. Through organization and planning of
language development we are controlling evaluation activities and tracking success
of the DSL. After each iteration, evaluation and development goals are scoped
and budget is xed in order to proceed to design and implementation activities
that are guided by (ii) Iterative User-Centered Design patterns. These support
engagement of evaluation expert into development process, that collects relevant
informations concerning the language engineer conception of problem solution
and users interpretation. Depending on artifacts to be evaluated (cognitive model
instances), abstraction layers that iteration focus on (abstract or concrete
syntaxes, integration environment, given semantics), evaluation methodology,
metrics de nition and relations used, each iteration evaluation design can vary a
lot. The third set of patterns, (iii) Experimental Evaluation Design supports the
de nition of evaluation process artifacts e.g. hypothesis, tests, metrics, samples
and statements [
By planning carefully the development process and organization of responsibilities and costs, the goal is to establish balanced management and engineering plan that will satisfy both: business and user needs, by optimizing impact of evaluation feedback on language development. Also, time that is invested into strategy and design of problem and its solution can be planned well with the technical implementation of solution.
By adopting UI evaluation process to DSL development [
When we consider GPLs, their users are part of population that master well
mathematical and technical concepts (e.g. they are highly skilled in
computations and logical reasoning). In order to develop programming solutions, GPL
users (i.e. programmers) need to understand also domain concepts. On the other
hand, as DSLs are meant to reduce use of computation concepts by putting focus
on the domain concepts. As such, DSLs are expected to be used by the much
diverse target population in di erent context of use [
Under the perspective of SLE, in order to experimentally evaluate a DSL, we
need to know what are the criteria involved, understand the notion of quality
from the relevant perspectives and understand the experimental process itself.
This complex challenge was covered by general model for DSL experimental
evaluation, i.e. [
Building DSLs is becoming very popular and by that there are increasing
needs of some pointers in topic of their cognitive congeniality to end user. The
value of contribution in this line of research is supported by the evidence of
the interesting return of investment on usability evaluation for other software
products [
Our goal is to evaluate our approach with respect application of approach to academical and industrial real cases. In particular, we address following research questions:
Q1: Can our approach identify usability problems early in development cycle? Q2: By using the proposed approach to language development, are we able to trace and validate design decisions?
Q3: By using proposed design of evaluation are we able to replicate experiments and reuse data and metrics?
Experiment planning. To evaluate rst research question (Q1), a set of evaluation experiments during DSL development that use presented technique should be conducted, where the main role of author is to be involved as evaluation expert (i.e. usability engineer). Comparison to baseline approaches to solving problem is expected to give us means of better quality of use for the produced solution. By comparing solutions within same domain, and iteratively comparing the design solutions from each of iterations we are planning to nd answers to our second and third research questions (Q2,Q3). By systematically comparing more DSL solutions within same domain, where each one cycle is focusing of di erent high level goals, we expect to have a means of reusing evaluation data, and building up statistical evidence of their validity and correlations. This will help validating our approach.
Threats to validity. In some evaluation cases it will be hard to distinguish if the success of produced DSL is to be due to our approach for usability or building a DSL by itself. However, we believe that is possible to overcome this threat by validating success of design decisions through di erent iterations and performing comparative analysis between them. It is needed to take a special care that for same context of experiment we use di erent subjects. Threat is that as there will be hard to nd a lot of expert users in domain, and often we will end up by having available just novice users for new problem solving approach. However, it is possible to validate di erent artifacts or abstraction layers with a same user groups. While performing evaluation experiments we need to be careful that we don't compare non-related things; like trying to compare DSL with Java (e.g. we should restrict the GPL just to the constructs that are related with a target domain: query procedures of Java with a query DSL). That is the reason why we need to precisely de ne scope and assure that context of use can t inside. It will be also challenging to evaluate iterative life-cycle. 7
Our analysis of target evaluation domain resulted in the conceptual models that
should be involved into the methodological framework (e.g. context model,
survey instruments and indicators, goal and requirements model etc.). In order to
support evaluation and language design, presented patterns are illustrated by
a real life case study of the usability evaluation of a DSL for the High-Energy
Physics [
Author is currently performing two evaluations of the approach. The rst one in industrial context that involves iterative development of a DSL for the humanitarian campaign ow speci cation (FlowSL). This example is expected to result in the detailed de nition of all artifacts that are needed to support development process. During the development process, FlowSL is being evaluated in comparison with baseline approach (hacking Ruby code), and comparative evaluations of iteration releases. In the end it is expected to be compared with BPMN with novice users, to assess learnability issues. Other experiment is run in context of summer schools on DSL development. Here the goal is to capture and validate design decisions and quality metrics between language solutions developed parallelly by di erent student groups.
An additional step is to conceptualize models for performing DSLs evaluation, metrics and traceability model of design changes and their impact. This support should be tailored to internal and external quality (e.g. syntactic and semantic models) and quality in use (e.g. interaction model) artifacts while using a DSL along several iterative evolution steps. It is expected to support more e cient performance of evaluation, and enable evaluator to explicitly model the process. This procedure will give us faster convergence of language development, as it enable us to monitor the impact of language evolution to the e ciency and e ectiveness of practitioners using the language (and its companion toolset).
As a side e ect, this evaluation work is expected to contribute to the validation of the claim that DSLs are more usable than GPLs. The impact of an evaluation process for DSLs could be interesting from an industry point of view. With many organizations developing their own languages, or hiring companies to develop such languages for them, this framework can aid them in reaching more usable languages.
Acknowledgments I gratefully thank to my supervisors Vasco Amaral and Miguel Goula~o . This work was partially supported by the CITI - PEst - OE /EEI /UI0527 /2011, Centro de Informtica e Tecnologias da Informao (CITI/FCT/UNL).