Common Variability Language (CVL) is a recent proposal for OMG's upcoming Variability Modeling standard. CVL models variability in terms of Model Fragments. Usability is a widely-recognized quality criterion essential to warranty the successful use of tools that put these ideas in practice. Facing the need of evaluating usability of CVL modeling tools, this paper presents a Usability Evaluation of CVL applied to a Modeling Tool for rmware code of Induction Hobs. This evaluation addresses the con guration, scoping and visualization facets. The evaluation involved the end users of the tool whom are engineers of our Induction Hob industrial partner. E ectiveness and e ciency results indicate that model con guration in terms of model fragment substitutions is intuitive enough but both scoping and visualization results suggest that CVL visual notation should be improved.
Common Variability Language (CVL) has been recently proposed by the
architectural board of the OMG as Variability Modeling standard [
Usability is a widely-recognized quality criterion essential to warranty the successful use of tools that put the above ideas in practice. This paper presents a usability evaluation of a Modeling Tool augmented with CVL (MT+CVL). The research question addressed by this evaluation is: Are Modeling Tools augmented with CVL intuitive enough to perform the main facets of variability modeling approaches (con guration, scoping and visualization)?
In order to materialize the ideas of CVL, we are going to use our industrial partner Modeling Tool, an induction hobs company that generates their induction hobs' rmwares following a model driven development approach. They used to follow a clone and own approach (without explicit de nition of variability) but we have augmented their modeling tool with CVL in order to model the variability existing among their products.
Our Usability Evaluation comprises both (1) test methods such as
Performance Measurement, Satisfaction Questionnaire and Interview and (2)
inspection methods such as KeyStroke-Level Model [
E ectiveness and e ciency results (con guration tasks 85% and 132.2%, scoping tasks 65% and 49.93%, visualization tasks 88% and 64.62%) indicate that model con guration in terms of model fragment substitutions is intuitive enough but both scoping and visualization require to improve the visual notation of CVL.
The remainder of this paper is structured as follows: Section 2 discusses related works. Section 3 summarizes the main concepts of the Common Variability Language. Section 4 presents an experimental study to evaluate the usability of the Modeling Tool with CVL. Then, Section 5 describes the results of evaluation and the set of Usability Problems detected. Finally we conclude the paper. 2
There are research e orts in literature towards the visualization of SPL
(Software Product Line) related artifacts. For instance in [
There is a concern in existing literature about the comprehensibility of feature
models and posible di culties for di erent user groups. For instance, in [
In [
There is also a concern about the usability of DSL and the tool used to
generate them. For instance, in [
CVL is a Domain Speci c Language (DSL) for modeling variability in any model
of any DSL based on Meta-Object Facility (MOF). The CVL proposal [
By means of the use replacement operation, users can perform substitutions, including fragments from the library into the model being edited. By means of the create replacement operation, users can create new replacement fragments and incorporate them into the library 1.
These are the main elements and operations of CVL, and need to be ful lled
to apply CVL for a given DSL. It is necessary to augment the DSL editor in
order to enable the operations de ned by CVL, but its application is the same
for any given DSL. For further details about the inner workings of CVL see [
InductionHob013
DSL Editor 1 Example of model fragment operations: http://folk.uio.no/oysteinh/demo1.htm
We have applied CVL to the modeling tool of our industrial partner. That is, we have augmented the Modeling Tool including and integrating the CVL operations and library (as presented in section 3), resulting in the MT+CVL that will be used through the rest of the study. This is the usual operation for augmenting an existing Modeling tool with CVL, and would be the same when applying to any other modeling tool.
Left part of Figure 1 shows the graphical editor of the models. Right part shows the replacements library, where all the replacements that are part of the MT+CVL are shown. In addition, the create replacement operation, enables engineers to create new replacements fragments that are included into the library. The Replace operation enables the engineers to substitute model elements from a product model (open in the editor) by replacements of the replacements library. 4.2
To evaluate the usability of the MT+CVL we had to know the main tasks that the end users perfom in the main facets of varibility modeling: Scope (in CVL, the creation and elimination of fragments), Con guration (the derivation of products) and Visualization (to make the user aware of the varibility). Six executable tasks were produced as output 2: T1 The induction hob IH013 has a problem with the module MOD008 and this module must be replaced by the module MOD014. In the other induction hobs the module MOD008 must not be replaced.
T2 The inverter INV016034 in the module MOD017 in the induction hob IH021 does not run correctly. The module must assemble the inverter INV019034.
This replacement must a ect every induction hob with the above module. T3 The induction hob IH021 in the module MOD073 has the inverter INV015034.
Its parameter is wrong. A new inverter must be created by cloning the wrong inverter. The new inverter has its parameter VMAX equal to 42. The replacement must a ect every induction hob with the above module.
T4 The module MOD021 in IH003 must replace the inverter INV015042 by
INV016042. This replacement must not a ect to other induction hobs. T5 To detect all components in the induction hob IH021.
T6 Which is the module most widely used of the set of modules (MOD021, MOD014, MOD017, MOD101)?
The tasks (T1) and (T2) are from con guration facet tasks, (T3) and (T4) are from scope facet tasks and, nally, (T5) and (T6) are from visualization facet tasks. 2 The identi cation of the components has been sanitized in order to preserve con dential information. However, omitted information is not relevant for the approach. 4.3
The Inspection Method (without users) chosen is Keystroke-Level Model. The
Keystroke-Level Model has two phases. The rst phase is to determine what
physical and mental steps a user performs to complete one or more tasks with
the CVL Modeling tool in order to predict the time that the user needs to do the
task. To do this, a duration is associated to each one of these actions or sequence
of operators (physical or mental), and then they are totaled. This duration is
calculated by using the average time that it takes a skilled user to complete the
action, as suggested by reference time values of [
The second phase is to analyze the above steps, looking for problems. Some
usability problems that the Keystroke-Level Model might reveal are that it takes
too many steps to perform a simple task, or it takes too long to perform the
task, or there is too much to learn about the interface, etc. [
The objectives of this phase are the assessment on Usability Measures and the
identi cation of usability problems. To achieve these objectives the following
UEM are used: Demographic Questionnaire, Performance Measurement,
Satisfaction Questionary and Interviews. These UEM are characterized by the
participation of the end users. The evaluation with users was as follows [
The evaluation involved the end users of the tool whom are engineers of
our Induction Hob industrial partner. The human computer interaction research
advises to use ve end users in the usability test to obtain 80% of the usability
problems [
In this UEM users performed a prede ned set of test tasks (see 4.2) while time and error data was collected. Quantitative data includes performance times, error rates, completed tasks or number of assistance. This data enables the calculation of e ciency and e ectiveness. Usability problems will come from the notes that the Evaluator has taken down during the test or extracted from an audio or video recording of the session.
Measures of e ectiveness take into account percent of right nished unassisted
tasks, percent of right assisted tasks, frequency of assists to the participant. The
e ciency value is the ratio between percent of right nished unassisted tasks
and the time to nish these tasks according to Common Industry Format (CIF)
for Usability Test Reports [
The values in Table 1 indicate that the most di cult or problematic tasks are the scope tasks. In contrast, the end users performed with great easy con guration tasks. On the oher hand, the end users performed correctly the visualization tasks, but it took them too much time taking into account the calculated values with Keystroke-Level Model (see Section 4.3).
Satisfaction Questionary After the performance measurement, a satisfaction questionary was lled by the end users. This questionnaire was System Usability Scale (SUS). SUS was used to determine user's subjective satisfaction with the SPL tool. Measuring user satisfaction provides a subjective usability metric. The questionnaire was composed by a ten questions with a Likert scale.
The data collected with SUS must be introduced in a spreadsheet to process
them. The global score was 73%, which shows that the end users classi ed the
CVL Modeling tool as \good", according to the scale suggested by [
The Interview Questions to perform this step had open questions and closed
questions. The closed questions were directed to check the understanding of the
tasks in the MT+CVL by the end users. For instance, the Instructor shown two
pictures to the end user with the state of the CVL Modeling tool after a task
and the end user had to choose wich picture is the correct. The open questions
aim was to detect the parts of the MT+CVL that were more problematic from
a usability point of view, along with the real causes of the problems [
We believe the results of the usability evaluation are relevant to model-based software developers, OMG variability standardization process and variability tools vendors as follows:
From the point of view of model-based software developers, as the case of our industrial partner, the usability evaluation results suggest that CVL can complement their current modelling tools to formalise and con gure variability (according to the result of E ectiveness and E ciency of variability tasks). The CVL library of model fragments turns out to enable them to shift from a Clone & Own approach to a systematic reuse of model fragments.
From the point of view of current OMGs variability standardization process this paper provides evidence that the current CVL proposal should be extended to provide a visual notation for the model fragment concepts. That is, current CVL proposal introduces the concepts of model placement and model replacement but the proposal lacks a concrete syntax to denote the model fragment boundaries. This lack of visual notation leads modellers to miss variation points in the models.
Finally, from the point of view of tool vendors, the usability evaluation results reveal that modellers require new editing capabilities to work with independent model fragments such as explicit creation, fragment comparison, fragment-based lters and propagations of changes.