Domain-Speci c Languages (DSLs) developers aim to narrow the gap between the level of abstraction used by domain users and the one provided by the DSL, in order to help taming the increased complexity of computer systems and real-world problems. The quality in use of a DSL is essential for its successful adoption. We illustrate how a usability evaluation process can be weaved into the development process of a concrete DSL - FlowSL - used for specifying humanitarian campaign processes lead by an international Non-Governmental Organization. FlowSL is being developed following an agile process using Model-Driven Development (MDD) tools, to cope with vague and poorly understood requirements in the beginning of the development process.
Domain-Speci c Languages (DSLs) and Models (DSMs) are used to raise the
level of abstraction, while at the same time narrowing down the design space [
DSLs are usually built by language developers in cooperation with domain
experts [
In this paper we apply action research to the development of a DSL, named FlowSL, designed to support managers in specifying and controlling the business processes supporting humanitarian campaigns. FlowSL is targeted to nonprogrammers. Their ability to use this language was identi ed as one of the highest concerns, so discovering usability issues in early development iterations, facilitated the achievement of an acceptable usability, while tracking the design decisions and their impact.
Usability has two complementary roles in design: as an attribute that must be
designed into the product, and as the highest level quality objective which should
be the overall objective of design [
This paper is organized as follows: Section 2 discusses related work; Section 3 provides a description of the evaluation approach; Section 4 discusses the language and evaluation goals and its development and evaluation plan; Section 5 discusses the lessons learned from the application of the described approach; nally, Section 6 concludes by highlighting lessons learnt and future work. 2
The need for assessing the impact of introducing a DSL in a development process
has been discussed in the literature, often with a focus on the business value that
DSL can bring (see, e.g. [
The term quality in use is often referred to more simply as usability [
Two important issues are how and when to assess DSL usability.
Concerning the how, we have argued that we can think of DSLs and their
supporting editors as communication interfaces between DSL users and a
computing platform, making DSL usability evaluation a special case of evaluating
User Interfaces (UIs) [
Concerning the when, we argued that we should adopt a systematic approach
to obtain a timely frequent usability feedback, while developing the DSL, to
better monitor its impact [
Building a DSL may have a rather exploratory nature, with respect to the DSL
requirements, particularly when the DSL is aimed for users with limited
computational skills or poorly understood, or evolving domains. To build up a
coste ective and high quality process, we de ned an agile and user centered DSL
evaluation process [
By placing DSL users as a focal point of DSLs' design and conception, the
goal was to ensure that the language satis es the user expectations. Besides
involving Domain Experts and Language Engineers, as typically happens in the
development of a DSL, we add the role of the Usability Engineer to the
development team. Usability engineers are professionals skilled in assessing and making
usability recommendations upon a given product (in this case, the DSL) and
gathering unbiased systematic feedback from stakeholders [
Each development iteration focuses on a di erent increment or level of abstraction to be evaluated or re ned. In the early phases it is important to study existing guidelines or standards for a particular domain and interview current or potential users about their current system or tools they are using to help them in accomplishing their tasks. This context of use study of a particular situation is intended to elicit the strengths and weaknesses of the baseline approach as well as the user expectations for the DSL.
Finally, once the language is deployed to users, an evaluation of its use in real contexts should be conducted, reusing the methods and metrics that were validated in the previous iterations.
The generic process described in the previous section was instantiated to the development of a concrete DSL | the FlowSL. FlowSL is a DSL for specifying humanitarian campaigns to be conducted by a non-governmental organization. FlowSL is integrated in MOVERCADO3 (MVC), a mobile-based messaging platform at the core of an ecosystem that enables real-time and a more e cient impact, by facilitating interactions among bene ciaries, health workers and facilities, e-money and mobile operators. The platform is meant to allow data mining in the search of insights that can be used to improve the e ects of the campaigns while supporting a high degree of transparency and accountability.
A rst version of the system (MVC1) was developed as a proof-of-concept to validate the key underlying principles. The second version of the system (MVC2) was developed in the form of a platform easily customizable by managers and extensible by developers of the organization's team. An important goal was to develop a language, FlowSL, to empower the Campaign Managers to de ne new kinds of campaign ows taking advantage of their domain knowledge.
Without FlowSL, managers needed to specify the ows orchestrating their
campaigns exclusively by means of presentations and verbal explanations. The
implementation and evolution of campaigns often resulted in rework and
unexpected behavior, usually due to vague speci cations, incorrect
interpretations, and di culties in validating the implementation, a phenomenon known
as impedance mismatch [
In order to balance the development e ort with e ective reusability (e.g. while
envisioning new marketing solutions), MVC2 was developed in a fast-paced way,
iteratively, along six two-weeks sprints, following an agile development process
based on Scrum4 and best practices of evolving reusable software systems [
The problem analysis was performed by mutual interaction and brainstorming between Domain Experts and Language Engineers in each sprint planning. Usability Engineers, in this case the researchers, had the role of observing and guiding the analysis outputs, while preparing the evaluation plan, without being directly involved in the language speci cation. To better understand and de ne 3 http://enter.movercado.org/ (accessed in July 19, 2014) 4 http://www.scrum.org/ (accessed in July 18, 2014) the problem, the required functionalities were described in terms of small user stories. Also, the new description of the user roles was introduced as the FlowSL is expected to change existing organizational work ows. To improve interaction between the development team and the users, all the produced results from the analysis were continuously documented in a wiki. As Scrum suggests, the project management was based on a product backlog maintained and shared on-line.
The relationship between the MVC system, FlowSL development, and relevant language users and expected work ow is presented in Fig.1. The original MVC1 system was developed in a GPL (Ruby). FlowSL was rst developed as a Ruby-based internal DSL. This approach allowed an optimal use of resources while keeping the existing system running. The second phase of language development was intended to support the managers to design the campaign ow speci cations by themselves, using simple and understandable visual language constructs. In the planned third phase (future work), the focus will be on evolving the language's editor to be collaborative and web-based. It will also be an opportunity to work on language's optimizations in the generation process.
After de ning the evaluation plan, the Usability Engineer prepared the usability requirements, using a goal-question-metric approach presented in Table 1, where goals conform to the Quality in Use model. These requirements were detailed and related to the right metrics and measurement instruments to perform appropriate usability tests in each development cycle. The validation of some of these requirements in earlier stages (e.g. understandability, readability) are stepping stones to achieve other soft requirements that cannot be evaluated in early phases (e.g. learnability). Multiple evaluations helped in validating and improving the set of identi ed metrics. Requirement Metric Understandability: Does the user NCon - number of concepts, NRel - number of relationships understand the di erent concepts NErrSpec - incorrect verbal de nitions of total NCon and Nrel and relations, and when and why given in language to use each one of the concepts? NErrMod - incorrect interpretations of presented NCon and NRel given in modeled solution Readability: How accurately is the NConInst - number of concept instances in the model ( ow), NReuser able to read the speci ed ows Inst - number of relationship instances in the model and interpret their meaning? NErrInst - number of incorrect verbal interpretation of NConInst and NRelInst given in language E ciency: How much time is TModInst - time necessary to read existing model instance ( ow) needed for a user to read existing TModSpec - time necessary to implement a new model instance or specify a new ow? ( ow) E ectiveness: Is the user able to NErrModInst - number of misinterpretation while reading existcorrectly implement a ow from a ing model instance ( ow) given high-level description of the NErrModSpec - number of errors while implementing new model required ow? instance ( ow) Learnability: How much time is TLearNov - training time necessary to learn novice users to use needed for users to learn the language TLearExp - training time necessary to learn domain FlowSL language? experts to use language Flexibility How long does it take to TModEvol - time necessary to evolve model instance ( ow) quickly change or correct existing TModCorr - time necessary to correct incorrect implementation ow speci cations? of model instance ( ow) Reusability How often user reuse NModReuse - number of reusing existing model instance ( ow) existing ow speci cations? NModEvol - number of evolving existing model instance ( ow) Expressiveness Is the user able to NErrCon - number of concept, or its property that user is missing specify all parts of ow? to implement model instance ( ow)
NErrRel - number of relationships, or its appropriate role that user is missing to implement model instance ( ow) Freadom of Risk Is the user able NEconDem - number of occurrence of economic damage due to to implement the speci cations in incorrect ow speci cation a way that can lead to unexpected NSofCor - number of occurrence of software corruption due to or unwanted system behavior? incorrect ow generation to system ( ow) Satisfaction How much is the user ConfLevel - self rated con dence score in a Likert scale satis ed with FlowSL? LikeLevel - self rated likability score in a Likert scale
5.1
The language goal of the rst iteration was to nd the di erences and commonalities in the Ruby code relevant for visual FlowSL and then do a corresponding mapping into a graphical representation, which would de ne the rst draft of the concrete visual syntax of FlowSL. This is considered as a way to describe appropriate activities step by step by mapping relevant fragments of extracted code to a visual representation and to identify repetitive patterns that represent reusable code artifacts. The evaluation goal was to assess whether this representation would be good enough to enhance the understandability and readability of ows from the perspective of Campaign Managers. It was expected that with the ow abstraction, the Domain Experts could describe more concrete requirements for the visual ow concepts.
The evaluation intervention was conducted when all existing ows of the MVC1 system were migrated to MVC2. This was the moment when the stakeholders could more clearly express the language purpose by distinguishing campaign processes from the ows underneath. The intervention was followed by an interview conducted with one representative subject : the Domain Expert with the role of Campaign Manager that was involved in specifying ows using the MVC1 system and who was also involved in the MVC2 Scrum development assuming, in that case, the role of Product Owner.
The evaluation document was prepared by the Usability Engineer containing 4 tasks: Task 1 and Task 2 describing user scenarios by roles and a global organization scenario that evaluator was asked to clarify and improve by placing him in organization work ow; Task 3 presenting alternative feature models of FlowSL that are reviewed and rede ned with a goal of separating campaign instantiation data and improving a vague language de nition; Task 4 presenting campaign ow based on simple and complex level of speci cation of the ow example (IPC Validation) that was found to be the most representative to describe. This task used metrics from the GQM table, which showed that the considered solution is very hard to understand.
The two major threats to validity of this evaluation were that it was subjective and only one user surrogate was involved. However, as the intended solution was seen as a step that helped to understand and to model the domain better, the guided interview helped to rede ne the technical concepts using domain terms. Evaluation resulted in a clearer plan for the next development cycles as well as clarifying usability requirements and appropriate tasks. The textual FlowSL makes explicit all relevant domain concepts, but also many extra more. considered more technical, The performed evaluation helped the DSL developers to adjust the level of abstraction to the needs of the DSL end users. The language at this phase, could be used by the System Managers (knowledgeable of the concepts of the baseline system), but not by Campaign Managers. 5.2
The language goal of this iteration was to develop a visual FlowSL prototype using the MetaEdit5 language workbench, that was selected for its support to top-down development. The evaluation's goal was to assess whether both the campaign managers and novice system managers were able to validate the specied ows using the newly proposed visual language and editor. These evaluations covered also the e ectiveness and expressiveness of the target language. 5 http://www.metacase.com/ (accessed in July 19, 2014)
The First evaluation intervention was organized very quickly and involved interviewing two subjects : the campaign manager from the rst development iteration and the system manager who was involved in the DSL development. The intervention consisted of one task where the subjects had the opportunity to compare two alternative concrete ow representations for the same ongoing example.
Based on the evaluation results the Usability Engineer produced designs of the concrete syntax for the DSL development team.
The second evaluation intervention involved the same subjects. The evaluation document had three tasks: Task 1 focused in assessing the understandability and expressiveness of the individual symbols; Tasks 2 and Task 3 meant to measure the readability and e ciency of the designed solution of the simple and complex ow. In addition to that, the Domain Expert was asked to describe the use of the symbols from Task 1 to produce the presented ow solutions and to describe the situations in which the existing ows can be reused. The evaluation session with the System Manager made it possible to identify important missing relationships between FlowSL concepts, as well as their connection points (hot spots) with the MVC system underneath.
For the third evaluation intervention the usability engineer introduced the design improvements motivated by the feedback obtained the previous evaluation. The new notations were designed and implemented, to be again compared. The tasks were similar to the previous intervention, although more elaborated. Here, the same subjects from the previous interventions were involved, as well as a member of the Scrum team.
For this third intervention the rules related to the usage of a certain activity were discussed. The usability engineer evaluated the cases where the system manager would have the need to hack the existing campaign ows, in order to customize certain functionality or rule. The goal was to use an example-based approach to identify improvements in the language.
It became clear that the evaluation materials prepared earlier helped to speed up the following evaluation phases and reduced their implementation costs. Besides, they became templates for the corresponding learning materials. Also, it was possible to abstract the language one level further, so that an online visual editor was built to support rapid high level speci cations of ows. To better deal with the increasing complexity of the speci ed models, rather than presenting all the concepts related to the ow de nition visually, a better option would be to present just high level concepts that are reused often, while others are hidden and based on prede ned rules that can be eventually recon gured textually. This approach empowered both the domain experts and the product owners to better control the design decisions. 6
In this paper, we presented an experience report on how to integrate top-down usability engineering practices into a bottom-up agile development of a DSL from its beginning. While playing the role of Usability Engineers, we experienced that small iterations involving Domain Experts, Product Owners and End Users can help us to clarify the meaning and the de nition of the relevant language concepts. This enables an early identi cation of possible language usability shortcomings and helps reshaping the DSL accordingly.
Early evaluations can be executed with a relatively low cost thanks to modeldriven tools that support production of rapid prototypes and presenting the idea. These evaluations support well-informed trade-o s among the strategy and design of the DSL under development, and its technical implementation, by improving communication. Besides, they improve the traceability of decisions, and of the solution progress. These iterations also help to capture and clarify contractual details of the most relevant language aspects that need to be considered during DSL development, and are a key element to improve the End Users experience while working with FlowSL.
We plan to validate our decisions, metrics, and the overall merit of the developed DSL, by performing experimental evaluations with both expert and novice users, by making comparisons to the baseline approach in Ruby, as well as to other process modelling languages that are natural candidates to serve for similar purposes (e.g. BPMN, JWL).
An additional step is to conceptualize the traceability model of design changes and evaluate its impact on the decision making process. We expect that in each iterative evaluation step we will not only identify opportunities to improve the usability of the DSL, but also to improve the evaluation process itself (e.g. through the validation, in this context, of the chosen metrics).
Weaving usability concerns into agile process is helping us to continuously evolve FlowSL, improving the cost-e ectiveness of DSL usage in specifying campaigns, and supporting a clearer assessment of which language concepts are more relevant to the di erent kinds of language users, which in turn helps nding the right level of abstraction and granularity of concepts. All these bene ts come with the cost of adding usability skills and of introducing new practices in the agile process, namely the introduction of lightweight metamodeling tools. The balance however, seems to be very positive, but ROI should be calculated precisely to support this claim.