-Multi-Level Modeling is receiving increasing levels of interest and its active research community is continuing to make progress. However, to advance the discipline effectively it is necessary to increase industry adoption and achieve better community cohesion. We believe that the key to addressing both these challenges is to promote the creation of more comparisons in the multi-level modeling field based on meaningful objective evaluations. In this position paper, we provide our view on what constitutes meaningful evaluations and discuss some of the issues involved in obtaining them, while presenting a broad overview of existing multi-level modeling evaluations. In particular, we emphasize the importance of understanding and managing the difference between internal and external qualities.
Although Multi-level Modeling (MLM) has seen steady
development over recent years, industry adoption is still virtually
non-existent (a rare application of MLM in an industry setting
is described in [
Another obstacle to the discipline’s future growth is the lack of research cohesion that may eventually make it impossible for community members to build on each other’s results. Without a sense of direction — i.e., a common understanding of the way forward – the discipline runs the risk of research diversification to a point where it loses its core focus and subsequently critical mass. We therefore believe that objective comparisons between competing approaches are not only desirable to provide a compass for future development but may eventually become necessary for the discipline’s survival.
Since convincing comparative evaluations are the key to addressing both of the aforementioned challenges, in this position paper we discuss some of the issues involved in performing such evaluations in the context of MLM. We first establish the basic parameters of meaningful, scientifically sound evaluations and then discuss a variety of concrete approaches, pointing to existing work where applicable.
The effectiveness of MLM evaluations in promoting industry adoption and research cohesion depends on the extent to which they measure something of relevance. Naturally, relevance itself depends on the stakeholders and their respective goals. However, in general, a meaningful evaluation provides results that have some kind of real-world relevance. In contrast, a meaningless evaluation – e.g., measuring the number of vowels in a language’s keywords – has no such real-world relevance. A comparison based on such an evaluation would not yield any meaningful insights into which language should be preferred for achieving any reasonable real-world impact.
Meaningful evaluations, on the other hand, should be designed to deliver some insights that provide the basis for pragmatic guidance. In order for an evaluation to be meaningful in our sense, it must address the following aspects: A1: Measurability Any targeted properties must be objectively observable. Ideally, measurements should yield numeric results that are directly proportional to the property being measured, as it is then not only possible to decide which approach is better but also by how much. It is never possible to judge an approach or tool to be, e.g., “good” or “productive”, without breaking down how the quality concerned manifests itself in terms of measurable properties.
A2: Conclusiveness Measurements should yield consistent results. Repeated performances of the evaluation need not yield exactly the same outcome, but they should deliver reliable values within a given margin of error. This also excludes results with low confidence (e.g., because they lack statistical significance).
A3: Impartiality The choice of the properties to be measured must not favor particularities of one solution that have no proven relationship to the ultimate goal. For example, a set of postulated requirements must be formulated in such a manner that they reference the problem-domain and the ultimate benefits to the targeted user rather than solution details.
A4: Trueness When using proxies (e.g. substitutes for realworld artifacts or practitioners) care must be taken to ensure that no circumstantial bias is introduced. Trueness, therefore comprises at least:
A4:1: Context Relevance Model proxies (i.e., samples used in lieu of real-world models) and the assumed operations on them should be demonstrated to be representative. Otherwise, a skewed selection could introduce undesired bias.
A4:2: Demographic Relevance Substitute users should be demonstrated to be representative of real users. In general, it is not possible to transfer results between different bodies of users (e.g., from students to practitioners in the field).
A5: Pragmatic Relevance Targeted properties must have a for MLM is its ability to reduce accidental complexity [
It is obviously challenging to “tick” all the above “boxes” measuring its size, that is, the number of its elements. For
in practice, but we feel it is useful to have a checklist that example, Gerbig performed a comparison based on model size
helps to document where an evaluation may be lacking. in his Ph.D. thesis using a sample model from the enterprise
architecture domain [
III. INTERNAL VERSUS EXTERNAL QUALITIES 50 modeling elements while the TLM version, using standard
Some of the aforementioned aspects are more difficult to workaround patterns such as the Type-Object pattern [
In our context, internal qualities pertain to the directly scenario [
Ultimately, only the external qualities have a direct bearing workaround techniques are responsible for increases of the size
on meaningful evaluations. However, due to the cost and of two-level models relative to their multi-level counterparts,
challenges involved in assessing external qualities directly de Lara et al. hence demonstrated that the observations made
in a meaningful way, one often attempts to approximate the in [
Interestingly, A1&A2 are most easily addressed by focusing to consider other classic metrics [
Aspects A3 & A5, on the other hand, are best addressed by overhead2 ((well-formedness rules + additional operations)=
focusing on the external qualities of an approach. External element count)) [
This weakness of classic metrics for evaluating MLM is IV. ASSESSING INTERNAL QUALITIES understandable given their motivation rooted in programming and/or modeling software. In these contexts, instances and
Complexity is one of the most commonly measured internal qualities since it is assumed to have a correlation with important external qualities such as maintainability, robustness, and trustworthiness etc. In fact, the main value proposition 1Apparently equivalent to the much debated “lines of code” metric for source code.
2Referred to as “complexity” in [
Demonstrate understanding of a model.
Read an incomplete model and correctly add missing parts.
Read a defective model and identify all issues.
Read a defective model and address all issues.
Create a model from scratch for a specified purpose.
discussion of the issues involved when aiming to perform meaningful evaluations while providing a broad overview of the MLM evaluations that have been conducted to date. The number of already existing MLM evaluations is encouraging and each of them represents a very useful step towards growing MLM as a discipline. However, our discussion has shown that the evaluations performed until now are overwhelmingly focused on internal rather than external qualities. Hence their pragmatic relevance – in the absence of the demonstration of their relationships are irrelevant to users. However, in many Another external quality which lends itself relatively domain modeling applications instances directly represent the straightforwardly to measurement is model robustness, i.e., the subject under study. In such contexts, the complexity of resilience of a model to user error. Here the goal would be to instance models is therefore very much a concern to users assess the likelihood of introducing errors when creating/mainand should thus be considered in evaluations. taining models. In particular, in the context of MLM to TLM
Instead of focusing on model properties (e.g., model com- comparisons, one would expect a two-level model to suffer
plexity), one may also consider language properties (e.g., lan- from more accidentally introduced errors than a
correspondguage expressiveness). For example, Atkinson et al. based their ing multi-level model. TLM would only provide the same
comparison of Melanee with MetaDepth on the differences be- safeguards against the introduction of model inconsistencies
tween their respective language features [
Ideally, feature-based comparisons should be accompanied used. The higher the adequacy of the language/tool, the better by an analysis of the impact of the different features on users. the modeler should perform when facing standard tasks. While certain features may seem elegant, ultimately their value To this end, we propose a “5C”-approach, comprising the must be assessed by considering external qualities. cognitive challenges listed in Table I.
Assessing the adequacy of an approach would be performed V. ASSESSING EXTERNAL QUALITIES by measuring completion speeds for representative concrete tasks of the above five kinds. If languages/tools actually yield different levels of productivity, one should expect to see differences in the C1-C5 completion measurements. Ideally, subjects should be chosen in such a way that results transfer to the intended user base in order to achieve demographic relevance. Full context relevance will be very hard to achieve with this approach as it is typically not feasible to work with realistically sized models in such experiments.
In order to evaluate the ultimate purpose of any approach
intended to deliver value to a user, it is necessary to determine
properties based on external qualities which relate to user
experience. As far as we are aware, only two MLM evaluations
of this kind have been performed to date. Both of these
investigate model changes and thus can be reasonably regarded
as evaluating (aspects of) maintainability. In his Ph.D. thesis,
Gerbig performed a comparative model change analysis by
counting the number of primitive change operations needed
to respond to certain requirements changes [
Kimura et al., also used a change-based approach to
compare Melanee, MetaDepth and EMF, with a particular
focus on extensibility [
It is natural that the first evaluations performed in an emerging field are focused on internal qualities, as these are usually much easier to asses than external ones. However, we believe that for a) the benefits of MLM to become convincing enough to generate serious interest from industry, and b) comparative evaluations to become useful enough to maintain the cohesion and momentum the research community requires, more user-oriented evaluations focusing on external qualities will be needed.
An important initiative in this regard is the “Bicycle Challenge” proposed by the MULTI 2017 workshop as a common sample scenario, allowing various MLM approaches to be compared based on an example with practical relevance.
Ideally, more such benchmarks will be designed in the future along with agreed upon usage scenarios, e.g., involving subsequent extensions, detecting and removing defects, etc.
It will remain a challenge to distinguish models and usage scenarios that have context relevance from those that do not, but any attempts to move MLM evaluations towards directly assessing external qualities or to strengthen the confidence in hitherto only assumed correlations between internal and external qualities will represent significant steps forward.