Model-based development of complex embedded systems commonly leverages multiple models to describe systems at di erent levels of abstraction and from di erent viewpoints. Inconsistencies between these models may cause delays or late changes throughout system design, development and maintenance. Therefore, an e cient model-based development practice requires support for managing consistency between di erent models. This is complicated in practice because these models are possibly created in several di erent modelling languages and modelling tools. Furthermore, in order to be suitable for industrial adoption, consistency checking support must t within current development processes and environments. This extended abstract describes a model-based development scenario from which a need for consistency checking emerges, followed by a discussion of requirements for consistency checking from a development process point of view. We also include a brief summary of early work on a prototype tool addressing these requirements.
In model-based development (MBD), models are used
as primary development artifacts in the speci cation,
design, and implementation of systems [
Complete consistency throughout development is
not possible and not desired, since that would only
hinder development [
Many of these approaches are demanding because
the user needs to de ne and maintain many complex
consistency de nitions. Furthermore, they may
require heavy changes to the development process and
thereby discourage industrial adoption. To promote
industry usage of a proposed consistency checking
approach it is imperative that its design considers this
process view [
In this extended abstract, we focus on what creating a consistency checking approach suitable for adoption in our target industrial contexts entails. We consider several aspects of the process view of consistency checking approaches and then summarize earlier results presenting a lightweight consistency checking approach in a speci c MBD scenario. Creating such an approach requires us to nd answers to the following research questions: 1. What are the types of inconsistencies users should be noti ed about? 2. What lightweight mechanism can detect the required inconsistencies? 3. How can this mechanism be made to t in with existing development processes and environments? 2
In this section, we describe a model-based development scenario and the accompanying challenge of keeping heterogeneous models consistent. Using a running example, we then discuss the di erent aspects of a consistency checking approach designed to overcome this challenge. 2.1
In this scenario, we consider the model-based
development of software for complex embedded systems.
Models are used in the design and speci cation of the
system and can be roughly divided into categories of
high-level and low-level models. High-level models,
e.g., SysML, AUTOSAR, or AADL models, describe
the system design or architecture. Low-level models
describe the software implementation, e.g., models in
Simulink or executable UML. In some cases,
development of the software implementation is done directly
in code. Therefore, we also consider the scenario of the
low-level models being code, e.g., C/C++. In general,
we can describe the high-level and low-level models
as pertaining to the system view and software view of
the system respectively. In this scenario, the high-level
model does not contain detailed enough information to
enable automatic generation of code or low-level
models. Models of both levels describe the same system
but do so at di erent levels of abstraction. It is said
that the low-level models are a re nement of the
highlevel models [
Adopting modern development practices, the models are developed in parallel and in short iterations. Work on high-level and low-level models thus happens in parallel and small increments. Typically, di erent engineers are responsible for the creation of the di erent models and they are possibly located in di erent teams and at di erent locations.
To illustrate the di erent aspects of the problem and to sketch the proposed approach towards a solution, we use a running example of a speci c MBD scenario throughout the remainder of this paper. Consider the development of the braking system of a car. As part of the system model, there will be a SysML description of the anti-lock braking system (ABS), expressed in structure diagrams. This part of the system is then further re ned and implemented in a Simulink model, which contains a detailed model for simulation and code generation. Now, consider the situation in which a design change to the ABS is made. In particular, we consider a change that makes the highlevel system model and the low-level (Simulink) model inconsistent, for example, because of a di erence in the assumed values for the car's total weight, or the amount of grip of the wheels. 2.2
We now describe the scope of the consistency checking challenge in the aforementioned MBD scenario. Multiple aspects of the challenge are considered. First, why inconsistency detection is needed and what inconsistencies an approach should be able to detect. Then, how inconsistencies can be detected. The method or mechanism by which inconsistencies are discovered is only a rst part of a complete approach towards managing inter-model consistency. An often-overlooked aspect of proposed consistency checking approaches in the literature is its place in the engineering process. When, by whom and where the consistency checks should be created, maintained, executed, and resolved in the development process are also important aspects to consider for an approach to be suitable for industrial adoption. Any consistency checking approach should be designed with also these aspects in mind. They are also each discussed in this section. 2.2.1
Consider a single high-level model and several low-level models re ning it. Let the models be consistent after iteration ih of the high-level model and iteration il of the low-level models. When the high-level model undergoes iteration ih + 1, it is likely that the lowlevel models, in il, are no longer consistent with it. For example, new functionality is added that is not yet implemented in low-level models, or existing function de nitions are changed, requiring updates to the lowlevel models. In our running example, the high-level model is the system model expressed in SysML.
Each of these inconsistencies should to some extent
be allowed, since disallowing inconsistency completely
is detrimental to development [
For example, consider a change in the high-level model requiring a number of changes and additions in low-level models. In our running example, this indeed happens, the system model is changed, requiring some changes in the Simulink model. The inconsistency is introduced immediately after the changes to the high-level model and should be tolerated, at least temporarily. This is important since the system design can be created ahead of the software design. They do not need to be created simultaneously, so forbidding them would hamper development. But at some point the inconsistency should be resolved in order to prevent propagation of inconsistencies through multiple models used to describe the system. For example, when required changes to one of the low-level models are not yet made and other low-level models start relying on this, now inconsistent, low-level model. In the running example, not updating the Simulink model of the ABS might result in its incorrect implementation. A consistency check would have shown the inconsistency between the models after the change to the highlevel model and in general, throughout development. It would also have given the developers a chance to resolve it before depending on the inconsistent low-level model and thus introducing more inconsistency. 2.2.2
In the described scenario, high-level models do not include low-level (software design) information such as algorithm descriptions. Rather, the similarities between the models at the di erent levels are structural. For example, part of the high-level model may de ne the structure of the required software, but with its most detailed elements being \black boxes", representing some particular functionality that is described in more detail in re ning low-level models. Thus, the models are related, since they describe the same system but at di erent levels of abstraction.
In this scenario, a type of inconsistency helpful to
know about is between model structures, i.e., the
lowlevel models should indeed contain representations of
what is described in the high-level models, even if that
description is just a black-box. For example, consider
a high-level model containing a de nition for several
software interfaces. Then, inconsistencies can exist
when the low-level model implements more than, less
than required, or di erent from the speci ed interface.
Note that a di erence can be seen as a combination of
implementing less than required (the correct thing is
missing) and more than required (the incorrect thing
is done in place of the correct thing). The type of
consistency checking in this setup is commonly referred to
as vertical inter-model consistency checking [
Another type of inconsistency regards shared values between models. In the running example, if the system model and Simulink model assume a di erent total mass of the car, this inconsistency could result in an incorrect ABS. Several other types of inconsistencies exist and could be relevant to detect, we plan an industrial empirical study to derive a set of most relevant consistency types and scenarios. 2.2.3
In the running example, a consistency check between the SysML model and the Simulink model could have detected the introduced inconsistency. Such a check should have been de ned earlier in the development process and be automatically evaluated upon changes to either model.
There are numerous published approaches that
allow for automatically detecting inconsistencies and
sometimes also automatically resolving them. A
considerable portion of work on consistency checking has
focused speci cally on UML models [
Essentially, detecting inconsistencies between two
di erent models requires two pieces of information.
The rst piece indicates the speci c model elements
that should be compared to each other and the second
piece de nes when those compared elements should be
considered consistent. When de ned in enough detail,
for example, by using Triple Graph Grammars [
Several approaches have aimed at automatically
resolving detected inconsistencies, e.g., in the context of
inconsistencies between UML diagrams [
Enabling the automatic detection of inconsistencies requires some initial de nition of what elements to check and what to consider consistent. Depending on the approach parts of the initial de nition could be made prior to development, or be implicit. For example, a logical rule could be de ned stating that each class in a UML class diagram must be represented in at least one UML sequence diagram in the same model. This rule implicitly contains the speci c mapping between model elements as each pair of class diagrams and sequence diagrams. When creating rules pertaining to speci c model elements, they can typically not be created completely in advance. The de nition of new consistency checks should therefore be possible throughout development.
During the system development, the design and implementation undergo numerous iterations, in each of which inconsistencies can be introduced and resolved. Furthermore, newly created model elements may incur the need for new or changed consistency checks. Therefore, the initial de nitions should be able to be maintained, i.e., changed, added to, or removed from. The purpose of this maintenance is then to sharpen consistency check de nitions and make their results more relevant.
Another aspect then concerns when in time to
execute the consistency checks. Although several
authors have argued for executing consistency checks
continuously (e.g. [
Finally, we must consider when to resolve identi ed inconsistencies. This is very much related to the time of detecting the inconsistencies, and the person responsible for resolving them. Furthermore, acceptable resolving times depend on the speci c project and the further development plans. In general, inconsistencies should be resolved as soon as possible, but depending on the exact situation, it may be acceptable to leave some inconsistencies exist for a longer time. In either case, this is up to the developers and an approach should appropriately support either choice.
In our running example, the de nition of consistency checks should have been done prior to this occurrence of the inconsistency (or it would not be detected). The execution should be done some time after the introduced change to the model. As argued above, there is a ne balance between too frequent and too infrequent checks. The resolution of the inconsistency is then up to the developer, that can now decide a best time to resolve the problem before it spreads to other components. 2.2.5
Apart from the time of de nition, maintenance and execution of consistency checks, of course an approach should consider which developers, in which roles, will create the checks. De ning sensible consistency checks requires knowledge of models at both abstraction levels. This implies the need for an approach providing these checks to be accessible to any involved developer in the project.
An important additional aspect concerns the responsibility of resolving detected inconsistencies. It is likely that an inconsistency caused by changes in the high-level should be resolved by changes in the lowlevel models. But this is not necessarily true. Changes in the high-level model might trail changes in the low-level models and therefore need updating. In the running example, consider instead that the Simulink model was updated rst to re ect a redesign of the system. Now it is inconsistent with the system model, but we would not want to revert the changes just made to the Simulink model in order for them to be again consistent. In either case, it is likely that the inconsistencies found by making changes at one level should be resolved at the other level and therefore by another engineer or group of engineers. In order not to raise more need for communication than it helps remove, a consistency checking approach should allow developers from both abstraction levels to view and understand results of consistency checks. 2.2.6
Similar to the when and who, an important aspect to consider is where in the development environment to de ne, maintain, and view results of executed consistency checks. To promote industrial adoption, a consistency checking approach should be accessible to all developers at all times throughout development. At the same time, it needs to t in with existing development processes and tooling environments.
Indeed, including a consistency checking approach within an existing complex tooling environment is not straightforward. Integrating a new separate tool, apart from the technical challenges of interacting with the other existing tools, risks never being looked at since it is not part of any existing development workow. Another placement of a consistency checking approach could be as a plug-in or extension to an existing modelling tool. While integrating an approach in one of the modelling tools has the advantage of developers using it without leaving their development environment, access to consistency checks is limited to users of that tool. This downside can be resolved by integrating consistency checks in a tool that is already part of the development process and accessible to all potential users, e.g., a version control system, an issue tracker, or a continuous integration (CI) server. 2.3
We aim for a pragmatic approach to consistency checking that is applicable in industrial practice. The essence of any approach is to notify developers of detected inconsistencies. Furthermore, an approach suitable in practice requires considering the process aspects of MBD in industry. Given the described MBD scenario in Section 2.1, a consistency checking approach is required that is generic, i.e. applicable to models conforming to various di erent modelling languages. Additionally, the consistency aspects discussed in Section 2.2 show the need for an approach to consistency management to be lightweight, i.e., easy to use and minimally interfering with existing development processes and environments. 3
In this section we describe previously published early
research results [
Our early results focus on checking vertical inter-model
consistency between heterogeneous models [
The two mappings are called a language consistency mapping (LCmap), and a model consistency mapping (M Cmap). A LCmap describes how meta-model elements of di erent modelling languages are mapped to a tree structure and what elements are not mapped to it. For example, subsystems in Simulink models are mapped to nodes in the tree but more detailed blocks, for example describing mathematical operations, are not. In this way, the LCmap is global. It is de ned once and reused for all consistency checks. A consistency check compares the trees created from models conforming to a particular language by invoking the corresponding LCmap. Note that the representation of models in the common format in comparison between common formats reduces the number of required mappings between metamodels. In a scenario where direct transformations exist between any two metamodels, for a total of n metamodels, n(n 1) (bi-directional) 2 transformations are required, whereas in this case, n su ce.
A M Cmap de nes the speci c model elements in speci c models between which consistency should be checked. Consider a SysML model, and a set of Simulink models re ning di erent parts of it. To create a consistency check for this case, the user selects a block in the SysML model and a Simulink model and indicates what type of consistency should be checked between them (re nement in this case). This constitutes the M Cmap.
A complete run of a consistency check then consists of the following steps: 1. A M Cmap between element e1 of model M1 in language L1 and a model element e2 of model M2 in language L2 is evaluated. (a) The LCmap between L1 and L2 consists of two de ned transformations, building tree representations T1 from M1 and T2 from M2. (b) Now, T1 and T2 contain e1 and e2 respectively. Let Te1 and Te2 be the subtrees starting at e1 and e2 respectively. 2. Now, a comparison algorithm is executed (depending on the choice between re nement and equivalence in M Cmap) (a) The algorithm compares Te1 and Te2. (b) The result is pass or fail. In case of fail a detailed description of the di erence between Te1 and Te2 causing the comparison algorithm to report a failure. 3. The user sees the result of all evaluated M Cmaps and can modify their de nitions, including options to mute the check (do not show unless the result changes) and skip the check (do not show until re-enabled). Of course, M Cmaps can also be deleted and new ones can be added.
The approach is implemented as a plug-in for Jenkins1, an automation server often used for CI pipelines. This is a design choice partly motivated by the requirements as outlined in Section 2. The placement in the CI pipeline allows for frequently de ning and maintaining of multiple M Cmaps. Furthermore, consistency checks are automatically executed after each integration, when models are typically internally consistent, thus avoiding very temporary and other uninteresting inconsistencies from being reported. Running checks at each integration seems like a reasonable frequency, not too frequent and therefore tedious but frequent enough to detect meaningful inconsistencies early. When already in place in a project, adding the plug-in to the CI pipeline entails minimal overhead. It is also a place independent of modelling tools and centrally accessible for viewing results and maintaining checks. Since our approach does not aim to automatically resolve detected inconsistencies but rather at making developers aware of them, it is important that results of checks are easily visible. 3.2
The approach presented in [
To make the approach even more lightweight, we envision using higher-order transformations (HoT) to derive the LCmap transformations from similar user actions as used to de ne the M Cmap. In that case, the user-input to LCmap de nitions would be limited to pointing to di erent modelling language concepts that can be compared to each other. The HoTs would then generate the model transformations required to represent the models in a tree, or other intermediate data structure. 3.3
As said, the main limitation of the proposed approach is that it aims at comparing the structures of models. When we consider the generalized case, in which we have found a comparison metamodel, a remaining limitation is the local view on consistency checking. In essence, our approach is rule-based and comparing speci c model elements.
Another fundamental limitation to our approach is that we limit ourselves to detecting inconsistencies, rather than attempting automatic resolution. We also do not yet automatically propose ways to x detected inconsistencies. Currently the limit of our approach is that it can provide a reason why the consistency check fails.
In terms of execution of the consistency checks, our approach assumes executing batches of checks at e.g. every integration. To limit the execution time, we aim to limit the amount of checks that are executed by running only those checks for which one of the involved models has changed since the last execution. Still, the approach is not checking for inconsistency in real-time, which is a design decision as explained in Section 2.
The strength of our approach with respect to existing approaches is mainly aimed to be that it is lightweight in usage. Where other approaches require complex rule de nitions, for example in rst-order logic or by means of bidirectional transformations, our approach requires of users only a model consistency mapping and a globally reusable language consistency mapping. Since the mainly considered scenario of applying this approach is in development and maintenance of complex systems, having a simple way of de ning consistency checks is imperative. 3.4
In addition to the three research questions posed in Section 1, the required evaluation can be summarized in the following questions.
1. Can the proposed consistency checking approach detect the required inconsistencies? 2. Is the proposed consistency checking approach suitable for industrial practice?
The rst question is commonly considered in
related work. Almost all approaches, including our own
as presented in [
Some evaluations of approaches consider part of this
process view. For example, Feldmann et al. [
An industrial evaluation of our approach requires a more mature tool and an implementation of the planned generalizations. The planned evaluation of our consistency checking approach then has two parts, corresponding to the two evaluation questions. First, we evaluate its suitability to detect the types of inconsistencies that should be detected. This evaluation is envisioned to be performed as a case study on a set of industrial models. The case study should show that the approach indeed detects the required inconsistencies and can do so in a reasonable amount of time, also when applied to models of scales commonly seen in industry. Second, an evaluation of the suitability of the approach in industrial practice is foreseen. Currently, we envision this evaluation as performing structured interviews with developers after they have been able to use our tool for some time.
As a rst step towards performing this evaluation, we plan to work closely together with our industrial partners to get an answer to the question posed in Section 2: \What types of consistency should be checked?" So what types of inconsistencies occur in practice between what types of artifacts. The second step is then to investigate di erent ways of de ning consistency checks and compare them with respect to the criteria we have speci ed here. An evaluation of the rst question can then be done by using the tool on realistic industrial models. The evaluation of the second question can be done either by means of user experiments, or by evaluating the features of the approach with respect to requirements as de ned by industrial practitioners, e.g. the requirement that the tool should be able to detect inconsistency between heterogeneous models. 4
Many published approaches consider the technical challenge of creating consistency checks with a focus on a speci c set of requirements, but largely ignore the practical aspects of de ning, maintaining, and executing them. In general, consistency checking work assumes the importance of checking consistency (why), explains the scope of the inconsistencies checked by their approach (what), and explains the mechanism devised to satisfy the approach (how). We have argued in this work the importance of additionally considering the engineers de ning and maintaining the checks as well as responding to their results (who), the placement of de nitions of the checks and execution in the development process (when), and the suitable place for tool support for de ning and creating them (where).
In particular, we have considered three research questions towards a consistency checking approach suitable for industrial adoption. These questions have been discussed in the context of a speci c MBD scenario. Further, early work was presented towards lightweight consistency checking. The approach requires only the selection of model elements between which to check consistency and then uses a global consistency mapping between the modelling languages to generate and execute checks. For now, the approach is targeted to a limited scenario in which consistency between the structure of models is checked. In order for our consistency checking approach to be applicable in practice, we need a more de nitive answer to the rst two research questions, generalize our proposed approach to answer the third, and evaluate that approach to verify that it is indeed bene cial to the development process.
This work is supported by Software Center.2