One of the key concepts of Model-Driven Engineering (MDE) is model transformations. Because of the crucial role of model transformations in MDE, they have to be treated in a similar way as traditional software artifacts. It is therefore necessary to de ne and assess their quality. In this paper we present a set of metrics for assessing the quality of ATL model transformations.
Model-Driven Engineering (MDE) is a software engineering discipline in which
models play a central role throughout the entire development process [
Our approach is aimed at assessing the internal quality of a model
transformation, i.e., the quality of the model transformation artifact itself [
The remainder of this paper is structured as follows. In Section 2, we analyze an ATL model transformation by hand. Section 3 describes the metrics that we have de ned for analyzing the quality of model transformations. In Section 4, we describe the tool we created that can be used for automatically extracting metrics from ATL transformations. Section 5 revisits the case we manually analyzed in Section 2 and presents an analysis of another case. In both cases, metrics are automatically extracted from the model transformation. We discuss related work in Section 6. Conclusions and directions for further research are given in Section 7. 2
In this section, we perform an analysis of a simple ATL model transformation by
hand. The transformation we chose for our analysis is the book to publication
transformation that can be found in the ATL zoo [
The model transformation takes one input model and returns one output model. It consists of one transformation rule. This rule is a matched rule and has no imperative section and has no local variable de nitions. Since there is only one rule in this model transformation and it is not abstract, there is no rule inheritance present. This rule has an input pattern consisting of one element and an output pattern also consisting of one element. The input pattern has a condition. This implies that not all model elements in the source model may be transformed. The output pattern has three bindings. One of the bindings is initialized by copying a model element of the source model. The other two bindings are initialized by the results of calls to helpers, i.e., the rule depends on two helpers. There are three helpers de ned in the transformation module. These are all the helpers in the transformation, since no other ATL units are imported in the transformation. All three helpers are operation helpers that are de ned in a context, module Book2Publication ; create OUT : Publication from IN : Book ; helper context Book ! Book def : getAuthors (): String = self . chapters -> collect (e | e. author ) -> asSet () -> iterate ( authorName ; acc : String = '' | acc + if acc = '' then authorName else ' and ' + authorName endif ); helper context Book ! Book def : getNbPages (): Integer = self . chapters -> collect (f | f. nbPages ) -> iterate ( pages ; acc : Integer = 0 | acc + pages ); helper context Book ! Book def : getSumPages () : Integer = self . chapters
-> collect (f | f. nbPages ). sum (); rule Book2Publication { from b : Book ! Book (b. getSumPages () > 2) to out : Publication ! Publication ( title <- b. title , authors <- b. getAuthors () , nbPages <- b. getSumPages () } )
Listing 1.1. Book to publication transformation in this case the same context. None of the three operation helpers take arguments. Also, in none of the helpers are local variables de ned. All helpers are used to manipulate collections. Two of the three helpers are used in the transformation. The helper getSumPages() is called twice and the helper getAuthors is called once. All of these calls come from the only transformation rule. The helper getNbPages() is never called by a rule or another helper. This may indicate that this helper is obsolete. In the documentation of the transformation is stated that the helpers getNbPages() and getSumPages() are two di erent implementations of the same functionality. In the helper getAuthors(), a conditional statement is used. This should be taken into consideration when testing the transformation.
For such a small example like this one, it is feasible to do an analysis by hand. However, when model transformations grow larger, it is infeasible to do this. Therefore it is desirable to perform an automated analysis. In Section 3, we describe a set of metrics that can be automatically extracted from ATL transformations by a tool that we describe in Section 4.
This section describes the metrics we de ned for assessing the quality of ATL model transformations. The metrics described here are speci c for ATL. However, for most of them a conceptually equivalent metric can be de ned for other model transformation formalisms as well.
The metrics can be divided into four categories, viz., rule metrics, helper metrics, dependency metrics, and miscellaneous metrics. In the remainder of this section, we will address each of these categories and elaborate on the metrics belonging to them. An overview of all the metrics can be found in Tables 1, and 2 in Section 5. 3.1
A measure for the size of a model transformation is the number of transformation rules it encompasses. In ATL, there are di erent types of rules, viz., matched rules, lazy matched rules, unique lazy matched rules, and called rules. In our metrics, we distinguish between these di erent types of rules. In case of a completely declarative model transformation, i.e., one with only non-lazy matched rules, it is to be expected that the amount of non-lazy matched rules is related to the size of the input metamodel, since typically a matched rule matches on one metamodel element. However, this is not necessarily the case, since matched rules can have input patterns that match on multiple metamodel elements at the same time or it may be the case that only part of the metamodel needs to be covered by the transformation.
Lazy matched rules and called rules need to be explicitly invoked in an ATL model transformation. Therefore it may be the case that there are lazy matched rules or called rules in an ATL model transformation that are never invoked. This can have a number of reasons, e.g., the rule has been replaced by another rule. To detect such unused rules, we included the metrics number of unused lazy matched rules and number of unused called rules.
ATL has support for rule inheritance. The use of inheritance may a ect the
quality of a model transformation in a similar way as it a ects object-oriented
software [
We have also de ned a number of metrics on the input and output patterns
of rules. The metric number of input pattern elements and number of output
pattern elements measure the size of respectively the input and the output pattern
of rules. Note that since called rules do not have an input pattern, the metric
number of input model elements does not include called rules. These two metrics
can be combined. The metric rule complexity change measures the amount of
output model elements that are generated per input model element. For
example, if the input pattern consists of one model element and two model elements
are generated the rule complexity change is 12 = 2. We do not consider model
elements that are generated within distinct foreach blocks, since the amount of
generated elements depends on the input model and can therefore not be
determined statically. This metric may be used for measuring the external quality [
The input pattern of a matched rule can be constrained by means of a lter condition. The metric number of rules with a lter condition measures the amount of rules that have such an input pattern. Using such lter conditions a rule matches only on a subset of the model elements de ned by the input pattern. Therefore it may be the case that there are multiple matched rules that match on the same input model elements. We de ned the metric number of matched rules per input pattern to measure this. Note that in general ATL does not allow multiple rules to match the same input elements, except in the case a rule overrides another rule.
To initialize target model elements, an output pattern has bindings. The metric number of bindings per output pattern is another measure for the size of the output pattern of a transformation rule. Typically, the bindings of an output pattern are initialized with attributes and references derived from elements in the input pattern. We propose to measure the number of unused input pattern elements to detect possibly obsolete input pattern elements. Matched rules require input pattern elements for the matching. In case that none of the input pattern elements of a lazy matched rule are used in that rule, this could be an indication that a called rule may be used instead. Note, however, that this is not always the case, since switching from a lazy matched rule to a called rule will no longer provide implicit tracing information that can be used elsewhere in the transformation. A metric related to the previous two is the number of direct copies. This metric measures the number of rules that copy (part of) an input model element to an output model element without changing any of the attributes. Note that this only occurs when the input metamodel and the output metamodel are the same.
Transformation rules can have local variables. These variables are often used to provide separation of concerns, i.e., to split the calculation of certain output bindings in orderly parts. This should increase the understandability of the rule. We de ne two metrics to measure the use of local variables in rules, viz., number of rules with a using clause and number of variables per using clause. We also measure the number of unused variables de ned in using clauses to detect obsolete variable de nitions.
ATL allows the de nition of imperative code in rules in a do block. This can be used to perform calculations that do not t the preferred declarative style of programming. To measure the use of imperative code in a transformation, we de ned two metrics, viz., number of rules with a do section and number of statements per do section. 3.2
Besides transformation rules, an ATL transformation also consists of helpers. Therefore the size of a model transformation is also in uenced by the number of helpers it includes. Two orthogonal distinctions can be made. On the one hand, there are helpers with context and helpers without context. On the other hand, there are attribute helpers and operation helpers. The operation helpers can be further subdivided into operation helpers with parameters and without parameters. Since helpers may be de ned in the transformation module or in library units, we also measure the number of helpers per unit. This gives an idea of the division of helpers among units.
Similarly to lazy matched rules and called rules, helpers need to be invoked explicitly. Therefore, again, it may be the case there are some helpers present in a model transformation that are never invoked. To detect such unused helpers, we included the metric number of unused helpers.
Helpers are identi ed by their name, context, and, in case of operation helpers, parameters. It is possible to overload helpers, i.e., de ne helpers with the same name but with a di erent context. To measure this kind of overloading we de ne the metrics number of overloaded helpers and number of helpers per helper name. Overloading is used to de ne similar operations on di erent datatypes. Of course it is also possible to de ne multiple di erent operations on the same datatype, i.e., in a di erent context. Therefore, we propose to measure the number of helpers per context.
To get more insight in the use of parameters by operation helpers, we propose to measure the number of parameters per operation helper. Also parameters may be unused for various reasons. To detect this, we propose the metric number of unused parameters per operation helper.
Helpers are often used to manipulate collections. Therefore, we measure the
number of operations on collections per helper. Also, conditions are often used
in helpers. The metric helper cyclomatic complexity is related to McCabe's
cyclomatic complexity [
We consider six categories of dependency, viz., units depending on other units, rules depending on rules, rules depending on helpers, helpers depending on helpers, helpers depending on called rules, and rules and helpers on built-in functions. To measure the rst three categories of dependencies we have de ned a number of fan-in, and fan-out metrics. For example, the metric number of calls to lazy matched rules per lazy matched rule measures the fan-in of lazy matched rules and the metric number of calls from helpers to helpers per helper measures the fan-out of helpers. An overview of all the fan-in and fan-out metrics can be found in Table 2 in Section 2.
The dependency of units on other units is measured by four metrics. The metrics number of imported units per unit and number of times a module is imported per unit are used to measure the import dependencies of units. On a lower level, the metrics number of calls from helpers in other units (unit fan-in) and number of calls to helpers in other units (unit fan-out) measure how the internals of units depend on each other.
The last dependency category, i.e., the dependency of rules and helpers on built-in functions is measured by the metric number of calls to built-in functions. Built-in functions are OCL functions and also additional ATL operations such as replaceAll(). There are some built-in functions that deserve special attention. First, there is the resolveTemp() function. This function is used to look-up references to non-default output elements of other rules. Therefore, it is to be expected that model transformations with a large number of calls to the resolveTemp() function are harder to understand. There are also the println() and the debug() function. The debug() function is used to print information to the console that can be used for debugging. In practice, we see that sometimes the println() function is used for a similar purpose. The occurrence of these two functions may indicate that the model transformation is still under development. 3.4
We de ned four more metrics that do not t the discussed categories. The metric number of units measures the amount of units that make up a model transformation. This metric can provide insight in the size and modularity of a model transformation. It may be the case that there are library units imported from which no helper is invoked in the transformation. To detect this, we de ne the metric number of unused units.
The last two metrics provide insight in the context of the model transformation. It is to be expected that model transformations involving more models are more complex. Therefore we propose to measure the number of input models and the number of output models.
We implemented a tool that enables automatic collection of the metrics
presented in Section 3 from ATL speci cations [
A T L p a r s e r
M e t r i c s e x t r a c t o r A T L m o d e l
M e t r i c s m o d e l
ule and possibly a number of libraries. The les containing this module and
libraries are parsed by the ATL parser. This results in ATL models representing
the model transformation. These models are the input for the metrics extractor.
This metrics extractor is itself a model transformation implemented in ATL. It
consists of one matched rule that matches on an ATL module, and a number
of lazy matched rules, each for calculating the value of one of the metrics. The
output of the metrics extractor is a model that contains the metrics data. This
model is used as input for a pretty printer. This pretty printer is a model-to-text
transformation implemented in Xpand [
In Section 3 we presented two types of metrics, viz., metrics that are measured over the entire transformation and metrics that are measured on a smaller scale, i.e., per unit, per rule, or per helper. The former type of metric has as single value for the entire transformation. We refer to this type of metrics as simple metrics. The latter type of metric has multiple values for the entire transformation, viz., one for every element that is measured (unit, rule, or helper). To assess the transformation as a whole we do not present all of these values. Instead, we give average, minimum, maximum, median and standard deviations for these metrics. We refer to this type of metrics as aggregated metrics.
The metrics extractor gathers data from an ATL model. Therefore, there is a
problem with identifying calls to helpers. The call to a helper consists of only its
name. However, a helper is de ned by its name, context, and parameters. In order
to distinguish between calls to helpers with the same name, more information
is required. Unfortunately, this information is only available during run-time.
We therefore decided to identify a call to a helper only by its name. We realize
that this is a threat to the validity of our measurements, since some of the
metrics may be calculated incorrectly. However, from our own experience and
by looking at the transformations available in the ATL zoo [
In Section 2, we performed a manual analysis of the book to publication transformation. In this section, we present the metric values that were automatically extracted from this transformation by our metrics extraction tool. Table 1 contains the simple metrics and Table 2 contains the aggregated metrics. The rst column of the tables states the category a metric belongs to. The second column contains the metric itself. The remainder of the columns contain the values for a metric. As a second example, we present the metrics we extracted from the metrics extractor described in Section 4 using that same tool. The extracted metrics are presented in Tables 3 and 4. Metric Value # Transformation Rules 90 # Matched Rules (Excluding Lazy Matched Rules) 1 # Matched Rules (Including Lazy Matched Rules) 90 # Lazy Matched Rules (Excluding Unique) 89 # Lazy Matched Rules (Including Unique) 89 # Unique Lazy Matched Rules 0 # Called Rules 0 s# Unused Lazy Matched Rules 1 irc# Unused Called Rules 0 te# Rules with a Filter Condition on the Input Pattern 0 em# Unused Input Pattern Elements 83 lu# Rules with a do Section 0 R# Direct Copies 0 # Abstract Transformation Rules 0 # Abstract Matched Rules 0 # Abstract Lazy Matched Rules 0 # Abstract Unique Lazy Matched Rules 0 # Rule Inheritance Trees 90 # Rules with a Using Clause 55 # Unused Variables De ned in Using Clauses 0 # Helpers 28 # Helpers with Context 11 # Helpers without Context 17 # Attribute Helpers 1 # Attribute Helpers with Context 0 # Attribute Helpers without Context 1 s # Operation Helpers 27 irc # Operation Helpers with Context 11 te # Operation Helpers without Context 16 rm # Operation Helpers with Parameters 13 ep # Operation Helpers without Parameters 14 le# Overloaded Helpers 0 H# Unused Helpers 0 # Unused Helpers with Context 0 # Unused Helpers without Context 0 # Unused Attribute Helpers 0 # Unused Operation Helpers 0 # Helpers with a Let Clause 14 # Unused Variables De ned in Let Clauses 0 # Calls to println() 7 .p# Calls to debug() 0 eD# Calls to resolveTemp() 0 # Calls To Built-In Functions (Built-In Function Fan-In) 519 # Units 1 .sc# Unused Units 0 iM# Input Models 1 # Output Models 1 The metric values give some insight into the transformation. There are 89 lazy matched rules and it can be derived from the metric values that all matched rules have one element per input pattern. Also, 83 input pattern elements are unused. From this we can conclude that 83 of the matched rules do not use their input pattern. This indicates that a large part of the lazy matched rules may be replaced by called rules if the use of the implicit tracing mechanism is not required. In this particular case it is possible to replace lazy matched rules by called rules. This is a point for future work.
There are seven calls to the println() function. In Section 3, we argued that this could indicate that debugging information is being printed. Here this is not the case. Instead, the calls to the println() function are used to generate warnings.
There are 90 rules and 28 helpers in one module. It may be advisable to introduce a library for the helpers to decrease the size of the transformation module.
From the values for the metric number of calls to helpers per helper, we can conclude that there are a lot of calls to a few helpers. These are calls to the helpers that calculate the aggregated metrics.
There is one unused lazy matched rule. This lazy rule is intended to calculate a metric that we considered as irrelevant. It should therefore be removed.
The advantages of using the tool to extract metric values are that it is fast, repeatable, and less error-prone. The disadvantage of using the tool is that it only provides numbers. Therefore, the analysis is not as detailed as the manual analysis we presented in Section 2. For example, from the metrics we can only conclude that there is one unused helper in the transformation, not which one it is. This is not a problem since we are interested in a relation between metrics and quality attributes. Moreover, the tool can easily be extended to provide more detailed reports such that it can be of assistance in locating problems. 6
In [
The authors of [
The authors of [
We have addressed the necessity for a methodology to analyze the quality of model transformations. In this paper, we de ned a set of 86 metrics for predicting the quality of model transformations created using ATL. These metrics can automatically be collected from ATL model transformation speci cations by a tool we created.
Metrics alone are not su cient to assess the quality of a model
transformation. They need to be related to quality attributes to understand which metrics
are relevant for assessing the di erent attributes of the quality model of
transformations. Therefore, we would like to conduct an empirical study in the same way
as described in [
We would like to thank Andres Vignaga for providing us with and letting us
use the metrics he de ned in [