=Paper=
{{Paper
|id=Vol-2245/commitmde_paper_1
|storemode=property
|title=Continuous Integration Support in Modeling Tools
|pdfUrl=https://ceur-ws.org/Vol-2245/commitmde_paper_1.pdf
|volume=Vol-2245
|authors=Robbert Jongeling,Jan Carlson,Antonio Cicchetti,Federico Ciccozzi
|dblpUrl=https://dblp.org/rec/conf/models/JongelingCCC18
}}
==Continuous Integration Support in Modeling Tools==
https://ceur-ws.org/Vol-2245/commitmde_paper_1.pdf
Continuous integration support in modeling tools
Robbert Jongeling, Jan Carlson, Antonio Cicchetti, Federico Ciccozzi
Mälardalen University, Västerås, Sweden
Email: {robbert.jongeling, jan.carlson, antonio.cicchetti, federico.ciccozzi}@mdh.se
ABSTRACT practices regarding the frequency and level (e.g. the entire system,
Continuous Integration (CI) and Model-Based Development (MBD) a sub-system or an own branch) of software integrations have been
have both been hailed as practices that improve the productivity of developed [19]. The terms Continuous Integration, Continuous
software development. Their combination has the potential to boost Deployment and Continuous Delivery are sometimes used inter-
productivity even more. The goal of our research is to identify im- changeably [26]. An unclear definition and interchangeable use of
pediments to realizing this combination in industrial collaborative the terms may lead to their devaluation [28]. Therefore, we refer to
modeling practices. In this paper, we examine certain specific fea- the definition of Continuous Integration, given by Fowler [11], as
tures of modeling tools that, due to their immaturity, may represent follows:
impediments to combining CI and MBD. To this end, we identify Definition 1. Continuous Integration is a collaborative develop-
features of modeling tools that are relevant to enabling CI practices ment practice where software engineers frequently, at least daily,
in MBD processes and we review modeling tools with respect to integrate their work into a shared repository. After each integration,
their level of support for each of these features. an automatic build is performed. On successful build, automated
tests are performed.
1 INTRODUCTION In MBD, this integrated work consists of models and other mod-
In this work we couple two concepts: Continuous Integration (CI) eling artefacts in addition to code. This may pose additional chal-
and Model-Based Development (MBD). CI refers to a subset of the lenges, such as differencing on model level, that are not encountered
Agile development practices as described by Martin Fowler [11]. when applying CI practices in conventional software development
Several empirical evaluations have shown the positive effects of CI projects.
on productivity in industrial software development projects [20, 27].
The MBD paradigm holds the promise of increased productivity of 1.2 Model-Based Development
development teams by raising the level of abstraction from code We use the term Model-Based Development (MBD) to denote mod-
to models [24]. The benefits of MBD on productivity have been eling practices in which models are used to capture functionality
empirically assessed in industrial settings too [5, 16, 21]. and possibly to generate code. Rios et al. distinguish five maturity
We hypothesize that CI practices can further improve the produc- levels of modeling practices [23]. The levels range from ad-hoc to
tivity of MBD. In our research, we aim at identifying impediments to ultimate and describe immature to complete modeling practices.
realizing these practices in industry. Eventually, although not in the Since our goal is to introduce CI practices in MBD, it does not make
scope of this paper, we aim at resolving them, thereby contributing sense to consider the first level, which describes immature modeling
to the maturity of collaborative modeling practices. practices where models are only used by individuals for e.g. design
In this work, we focus on technical challenges towards introduc- or documentation. Instead, we are interested the more advanced
ing CI practices in MBD. We examine modeling tools to identify levels of modeling practices where models are used by multiple
particular features that are commonly underdeveloped and thereby team members and the eventual code or application derived from
may represent potential impediments to combining CI and MBD. the models must be consistent with these models.
In particular, we answer the following research questions:
(1) What are relevant features for a modeling tool to be able 2 RELATED WORK
to support CI practices? Since the publication of the agile manifesto in 2001, research on
(2) To what extent are these features provided in current mod- combining Agile and modeling practices [4], has been performed.
eling tools? Evidence-based software engineering studies of the field have shown
In the remainder of this introduction, the concepts of CI and that Agile modeling is not a mature field yet [1, 15]. In particular,
MBD are described in more detail. The rest of the paper is organized these studies identify the need for more reports on Agile modeling
as follows. Section 2 outlines related reviews of modeling tools. practices in industry.
The first research question is answered in Section 3. In Section 4, Although Agile modeling is a topic treated in several research
the second research question is answered. The results are discussed articles and CI is named as a needed practice in modeling [2], we
in Section 5. Threats to the validity of our work are outlined in have only found few that go into the details of the Agile practice
Section 6 and the conclusions are provided in Section 7. of CI in combination with MBD. Garcı́a-Dı́az et al. identify four
problems in applying CI practices in an MBD project [14]. Among
1.1 Continuous Integration these problems, the two most interesting in relation to CI (as we
CI is one of the twelve Extreme Programming (XP) practices [11]. will see later in Section 3) are version control systems for models
In turn, XP is one of the elements of the software development and incremental code generation. Additionally, they stress the im-
concepts published in the Agile manifesto [6]. Since then, various portance of uniform, user-friendly interfaces and the variability
of technologies in different phases of the pipeline. They build and 3.1 Core CI Aspects
evaluate a prototype solution to resolve these identified problems In their literature study, Shahin et al. provide an overview of
in an MBD project. This solution focuses on modeling approaches types of tools used to form a pipeline for Continuous Deployment
where models are used to generate 100% of the code for an applica- (CD) [26]. The categories are: Version Control System, Code Manage-
tion, whereas we consider also modeling approaches where only ment and Analysis, Build System, CI Server, Testing, Configuration
parts of the code are generated. and Provisioning, and CD Server. They note that an implementation
Some work has been done towards a build server for MBD [30]. of CD does not necessarily include all categories. Furthermore, the
The authors identify the need to support verification and validation set of CI practices can be considered a part of that CD pipeline,
of models. Furthermore, they argue the need for build tooling to where the latter two tool categories (i.e. Configuration and Provi-
support a mix of automatic and manual actions. sioning and CD Server) are excluded. Nevertheless, the CD pipeline
Recently, early work towards resolving impediments to combin- is a good starting point to find out relevant features for combining
ing Continuous Delivery and MBD was presented [12, 13]. The CI and MBD.
authors identify the main technical impediment to be the model- We now elaborate on each of the proposed categories and their
awareness of the integration server. Furthermore, they remark that relevance for CI practices in MBD. A Version Control System (VCS)
for all typical MBD activities, tools that can be included in a pipeline is used to manage different versions of developed artefacts. The
are available. Our approach looks at this from a different perspec- artefacts are typically stored in a shared repository, sometimes
tive; we explore modeling tools and then consider the possibilities allowing developers to copy it and work on their own instance, or
to introduce CI practices. branch. Integration of code is done into this shared repository. Code
Naturally, there are aspects of MBD in general, not just the com- management and analysis techniques such as static code (or model)
bination of CI and MBD, that are relevant to its adoption in industry. analysis might be employed to improve the quality of artefacts in a
The impediments may be technical or be related to organizational CI process. They are however themselves not closely related to the
factors of the software development process. Tooling is one of three main activities in the definition of CI: integrating, building
the aspects preventing a more wide-spread adoption of MBD in and testing. Therefore, we do not include the Code Management and
industry; another is the lack of clear processes to support devel- Analysis category. A build system typically combines artefacts to
opment [33]. There are numerous organizational challenges that, create executables. In the case of MBD, this could mean generating
if not properly tackled, hinder agility in MBD [29]. Other impedi- code from models but also keeping the consistency of several related
ments to the adoption of MBD that are highlighted in literature are models. CI servers do not perform builds themselves but rather
the lack of tool interoperability [18], and the steep learning curve execute builds and automated tests in other tools, the results are
for developers [5, 29]. Furthermore, impediments might lie in devel- combined in a status overview [11]. These automated tests are
opment processes of companies and the required time and money important to indicate the quality of integrations. In that way, they
investments to change these. Technical challenges include poor contribute to increasing the predictability of the amount of work
scalability of the modeling practice in general in large industrial left, one of the purposes of CI.
applications of MBD [5]. Related to these are automated test gener-
ation and performance of generated code. Finally, Selic mentions 3.2 MBD Aspects Related to the Core
challenges regarding the integration with legacy systems, trace- Continuous Integration CI Aspects
ability of generated code and the ability to execute models [25].
Nevertheless, we limit the scope of this paper to those aspects We differentiate between three types of projects to which CI can be
relevant for introducing CI practices in MBD. applied. The first type is traditional software development, no mod-
In megamodeling, a model is created to describe the relationships els are used at all. The second type is very mature MBD, where all
between all concepts in a modeling project [10]. Since part of the code is generated from models and no manual coding is performed.
work to enable CI practices is to chart the artefacts in a project and The third type concerns less mature kinds of MBD, where code is
how they are related, a CI pipeline could be seen as a megamodel. partially generated and then manually extended to form a complete
In this paper, we do not further explore this relation and instead application. We consider the latter two types in this paper. The
focus on existing modeling tools that are used in current practice. inclusion of modeling artefacts in the CI process requires that some
parts of integration, testing and building are handled differently. In
this section, we identify the specific aspects of MBD that constitute
these differences.
3.2.1 Integration. We consider the integration of changes to
individual artefacts into the repository and focus on the aspects
3 IDENTIFYING RELEVANT ASPECTS of model differencing and model merging to facilitate this inte-
In this section, we identify aspects of modeling tools that are rele- gration. Integration is thus considered an activity localized to the
vant for enabling CI practices in MBD. We first identify core aspects directly changed artefacts. If multiple artefacts need to be changed
of CI based on our definition and existing literature. Then, we list to maintain consistency of the models, this is considered as the
particular aspects that realize these general CI aspects in the MBD responsibility of the developer.
domain, based on existing MBD literature. We submitted the identi- In order to integrate models in a shared repository, there must
fied aspects for review to two industrial practitioners of MBD. The be a way to control their different versions. Zhang and Patel [34]
resulting aspects are summarized in Table 1. refer to CI in MBD as “Continuous Modeling.” They identify the
need to merge frequently, but also note that merge tooling cannot analogous to traditional software development in which e.g. non-
handle many simultaneous changes. Merging of models is also compiling or incorrect code is integrated. In such cases, the builds
identified as an important aspect to the adoption of modeling tools or tests are expected to fail.
in general [7, 29]. Alternatively, pessimistic locking is used to avoid
merge conflicts by allowing only one developer at a time to make
changes to a model or part of a model [3]. 3.2.4 Model-Awareness. In some work about combining CI and
MBD, authors have argued the need for more model-awareness in
tasks related to CI. In case of the build tooling, it is argued that
3.2.2 Building. There is no direct MBD equivalent of a build sys-
manual actions that are typically performed in MBD should be
tem for conventional programming languages. A build system for
taken into account and that testing should focus on validation and
models requires more steps than a build system for code, since code
verification of models [30]. Others argue that the entire pipeline
needs first to be generated from the models and possibly altered or
should be model-aware, such that dependencies between artefacts
completed by developers. Given the scope of our research, we adapt
and between jobs in the pipeline that would be lost in textual
the previously identified aspect of a build server to include more
representations can be discovered [13]. Garcı́a-Dı́az et al. also note
model-specific actions. Automated code generation is a central part
lack of model-awareness, particularly in version control systems
of continuous integration in a modeling context [34]. Furthermore,
and code generation [14]. We do not add an aspect model-awareness
it is argued that code generators should work incrementally, i.e.,
to our list but when discussing the other aspects in modeling tools,
that code should be generated only for parts of the model that have
we do take into account to what extent their implementation is
changed [14]. But the key elements of building in MBD are the
model-aware.
ability to generate code and the ability to synchronize models and
The need for model-awareness may also refer to the need to syn-
code. Therefore, we add only code generation and model discovery
chronize several models when one of them is changed. In our case,
as relevant aspects.
this model synchronization is limited to the consistency of models
For the different types of CI projects, these aspects can have
and code, and between several models in a single project. In other
different meanings. In projects with complete code generation, this
cases, the term co-evolution is used to refer to similar activities that
generation is a task for the build server and is not performed locally
also include the synchronization of models and metamodels, or the
by developers. When code is only partially generated, this can be
synchronization of models and model transformations. Since the
done both locally and on the build server. In case of complete code
most used metamodels in the considered tools are UML and SysML,
generation, model discovery is irrelevant, it will never be done since
they and the model-to-text transformations (code generators) are
the code is never manually edited. Conversely, in a partial code
typically not changed during a project. We therefore refer to this
generation scenario, model discovery can be needed both locally
MBD aspect as “model synchronization.”
and remotely. In the simplest form, a developer locally makes
Extensive support for activities related to model synchroniza-
changes to a piece of generated code and immediately updates the
tion, such as automatically handling inconsistencies, is required
corresponding model too. In a more complex scenario, a developer
in modeling tools [32]. Model synchronization is also related to
could only change the code, integrate it and expect the build server
code generation and model discovery, i.e., the automatic creation
to propagate her changes to the model.
of a model from code. Since the generated code and models can
become inconsistent after changes to either. Modeling tools can
3.2.3 Testing. We distinguish three components of testing in support this synchronization, e.g. by providing automated impact
MBD: validating the model with respect to syntax, verifying the analysis for changed artefacts, but the process cannot always be
model with respect to predefined requirements and verifying mod- automated. Therefore, manual actions could be required during
els after integration (integration testing). The aforementioned con- model synchronization, this step is unsuitable for inclusion in au-
tinuous modeling practices specify unit and integration testing as tomated builds. Since we envision a CI pipeline for models that is
important practices to build confidence in the product [34]. Veri- automated similarly to that for traditional software development,
fication and validation of models are identified as crucial features we consider model synchronization to be a task performed locally
of a build server for MBD [30]. In both cases, “testing” specifically by a developer. Consequently, the CI server or build server is not
refers to testing the correctness of the created models, it is assumed concerned with tasks such as propagating changes to other arte-
that correct models yield correct code and thus correct applica- facts. The identified need for support is still relevant, since the
tions. We therefore add model validation and verification, as well developer should be supported in her local work.
as integration testing, as sub-categories of the testing aspect.
Again, some distinctions can be made in testing between the
different types of CI projects. In case of complete code generation, 3.2.5 Automation. Interoperability of a modeling tool with other
testing the models and the generated code should yield the same re- tools is an important aspect to consider too [34]. To assess their
sults. In partial code generation projects, code is the predominantly suitability to cooperate with CI servers, we look at possibilities to
tested artefact. In both cases, validation of models with respect to run modeling tools in batch mode or call their functionality from
syntax is an implicit part of the code generation process, which the command line. If this functionality exists, it can be used to
will not yield correct output for invalid models. This validation can create a script that automates part of the CI pipeline, including
also be a local action, but this is not required for the CI process. building and testing. Such automation is of crucial importance to
It does not prevent the integration of invalid or incorrect models, the adoption of CI practices in industry.
3.2.6 Summary. The aspects identified in this section are sum- UML and SysML, among the most used modeling languages in in-
marized in Table 1. It contains primary aspects (in bold) and sec- dustry [18]. Exceptions to this are BridgePoint, which supports
ondary, more specific aspects. In Section 4, we evaluate a set of xtUML (an executable dialect of UML), LabVIEW, which supports
modeling tools with respect to these primary aspects based on their their “G” graphical modeling language, and Simulink, which sup-
support of the secondary aspects. Notably, not all CI aspects are ports modeling in the Simulink language. Most of the tools thus
directly mapped to a single MBD aspect. Rather, the MBD aspects support general purpose modeling languages. The most advanced
are specific to their domain and target a more specific function- modeling practice includes the creation of custom Domain-Specific
ality than the general CI aspects. In the table, the citations refer Languages (DSLs). Tools used for that purpose are further away
to literature sources used to identify the relevance of the related from the state of practice at our industrial partners and therefore
aspects. not included in this evaluation. An overview of the considered tools
is shown in Table 2.
Table 1: Identified relevant aspects of CI in MBD.
Table 2: Evaluated tools in this review.
CI MBD
Supported
Integration [26] [7, 29, 34]
Modeling
Model Differencing
Tool Vendor Languages
Model Merging
Building [26] BridgePoint OneFact xtUML
Code Generation [14, 34] Enterprise Architect Sparx Systems UML + SysML
Model Discovery [32] Integrity Modeler PTC UML + SysML
Aspects
Model Synchronization [12, 32] LabVIEW National Instruments G
Testing [26] Magic Draw No Magic UML + SysML
Model V&V [30] Papyrus Eclipse UML + SysML
Integration Testing [34] Rhapsody IBM UML + SysML
Automation [26] [12] Simulink MathWorks Simulink
Building
Testing
4.2 Other Tools
In addition to the modeling tools, a CI pipeline typically also in-
4 SUPPORTED ASPECTS volves Version Control Systems (VCSs) and CI servers. There exist
In this section, we introduce the evaluated modeling tools and numerous open-source and commercial CI servers, such as Jenk-
discuss for each tool how it implements support for the primary ins, Travis, and TeamCity. Some of these allow for a completely
aspects in Table 1. We discuss the tools with respect to their support custom defined pipeline whereas other tools provide users with a
for the relevant aspects of CI in MBD as discussed in Section 3. choice between predefined pipelines for some programming lan-
Note that the selection of modeling tool(s) depends on more than guages. Since we aim at using these tools in MBD processes, we are
just the ability to use it in a CI process applied to an MBD project. particularly interested in those CI servers that allow the definition
Conversely, a CI process in MBD depends on more than just the of a custom pipeline. Therefore, we will mainly refer to Jenkins in
used modeling tool(s), such as the maturity level of the modeling the remainder of this section.
practices. The goal of our work is not to identify the best modeling
tool for CI, but rather to investigate what impediments in applying 4.3 Tool Evaluations
CI to MBD projects exist. We evaluated how the selected tools support the primary aspects
depicted in Table 1. The evaluations are based on publicly available
4.1 Modeling Tools documentation and research papers about the tools. The results of
The selection of modeling tools was based on their use in industry the evaluation are summarized at the end of this section in Table 3.
and on inputs from our industrial partners. We included the four
most-used1 tools in industry as reported by practitioners [18]: Mat- 4.3.1 Integration. In BridgePoint, version control is difficult
lab Simulink, Sparx Systems Enterprise Architect, IBM Ra- to achieve [31]. Automated merging is not supported but the tool
tional Rhapsody2 , and National Instruments LabVIEW. After does show a visual difference between two versions of a model.
discussions with our industrial partners, this list was supplemented This is not necessarily an impediment to introducing CI practices,
with four additional tools that are most relevant to their daily but in practice it may discourage developers to integrate frequently
work: NoMagic Magic Draw, PTC Integrity Modeler, OneFact if every integration potentially requires a large manual effort.
BridgePoint, and Eclipse Papyrus. Most of these tools support Enterprise Architect (EA) has no integrated support for model
versioning but relies on pessimistic locking of packages (parts of
1 Excluding Eclipse-based tools and in-house tools, since they cannot be specified to a
models). This system grants user exclusive editing rights on a
particular tool. They are reported as second and fourth most used respectively [18].
2 In [18] the reported tool is Rational Modeler, but we include Rational Rhapsody, package, thus preventing conflicts due to simultaneous changes.
which can be seen as its successor. The tool does support the integration of several third-party version
control systems, which can be used to store and manage the history a VCS, is not feasible because the VCS is typically not model-aware,
of EA models. Additionally, LemonTree is a third-party project which is required for merging at model level. Furthermore, line
that supports optimistic locking, three-way merging and branching based differencing of the XML representations is not appropriate
for EA models. for models [3]. The third and most feasible versioning approach
Integrity Modeler contains a built-in service for configuration when introducing CI practices is to enable the integration of a
management. It includes a weak optimistic locking mechanism version control tool in the modeling tool, but circumscribing model
allowing multiple developers to collaborate on the same artefacts differencing and merging to the modeling tool itself. Indeed, this
simultaneously. When multiple users are editing the same artefact, level of support is provided by multiple tools: Simulink, Rhapsody,
the changes of one of them are visible to each of the others in Magic Draw, and Papyrus.
real-time. Alternatively, there is an optimistic locking mechanism
available, where these changes are not visible. Then, merges can 4.3.2 Building. BridgePoint provides integrated support for
be performed automatically and their results manually edited to code generation. This functionality forms the “Translatable” part in
resolve merge conflicts. “eXecutable Translatable UML” (xtUML). Models in xtUML can be
LabVIEW includes a tool showing graphical differences between transformed to C, SystemC or C++ using included model compilers.
model versions. VCSs, such as Git and SVN, can be used to keep These compilers are open source and can be customized. It is also
track of different model versions. The differencing functionality of possible to create new compilers to translate models into differ-
those tools can then be redirected to use the graphical difference ent programming languages. Changes to generated code are not
available in LabVIEW. Merges can be performed automatically and propagated back to models. So, there is only support for one-way
merge conflicts can be resolved manually. development and not for the round-trip from models to code and
Magic Draw contains a built-in server for version control, it back. When the generated code is a complete application rather
provides a model repository and supports collaboration through than a skeleton or a detached, individual subsystem, this is not
branching and merging. Branching allows multiple developers necessarily an impediment to introducing CI practices.
to work in parallel on the same project. A plug-in is available In Enterprise Architect, skeleton code can be generated from
to support merging at model level. In case branching is not used, both Class diagrams and Interface models. More detailed code can
concurrent changes directly on the mainline are prevented by means be generated from sequence-, activity-, and state machine diagrams.
of pessimistic locking. The locks can be acquired at sub-model level, Several languages are supported, including C, C++, C#, and Java.
i.e., parts of a model can be locked for editing. The locks can then Reverse engineering is also (partially) supported since some UML
be released or maintained on commit. diagrams can be generated from code. Enterprise Architect
Papyrus can be extended using plug-ins that are part of the includes a development environment where generated code can
Eclipse Modeling Framework (EMF). The Collaborative Modeling be edited. This environment also supports typical functionalities
initiative provides such plug-ins, in terms of collaboration support of a code editor such as debugging and profiling. Code generation
for modeling in Eclipse, by using EMFCompare for the detection and reverse engineering can be combined and an option exists
and merging of changes, EGit for distributed version control and to keep models and code synchronized. When generated code
Gerrit for reviews of models. This allows developers to create a is updated due to a change in the model, the body of methods
branch for a project, make changes and merge them into the main- is untouched, only their headers are changed such that previous
line while staying on the model level. The included version control work is not undone. Although Enterprise Architect provides
system EGit is an implementation of Git, incorporated in Eclipse. traceability matrices from requirements to models for requirements
EMFCompare shows differences of changes between model ver- engineering, impact analysis for changes in models is not supported.
sions from several views (graphical, textual, tabular). It can au- Some work has been done on creating impact analysis techniques
tomatically merge changes, or in case of conflicts in three-way for any Enterprise Architect model, showing the potential of
merges, allows the developer to choose the version to be integrated. third party solutions to solve this problem [17].
Rhapsody includes the tool DiffMerge, which can show graph- Integrity Modeler supports code generation from class and
ical differences and automatically merge models or projects con- state-machine diagrams in several programming languages, includ-
taining models. In case of any merge conflicts, the developer is ing Ada, C++, and Java. The generated code might be complete but
shown a graphical difference between the versions and can resolve usually manual editing is required [22]. The tool includes function-
the conflict by choosing one of the versions. The tool also supports ality to keep models and code synchronized in real-time when man-
integration of version control systems ClearCase and SVN. ually altering the code. Furthermore, it supports impact analysis
Simulink provides version control support through an integrated by letting the user define relationships between different modeling
SVN instance but can also be used together with Git. This allows artefacts. These relationships can then be visualized to identify
a project to be branched and thus models to be edited in parallel. potential model elements that need to be synchronized.
Simulink contains an integrated tool for three-way model merging. Using additional code generators, C and Ada code can be gener-
The tool automatically merges models and on conflict offers a choice ated from LabVIEW models. Templates on which the generations
between the remote, base and local change. are based can be customized. Alternatively, the generated code
can be customized after generation. Code generation is a one-way
Summary. There are three main approaches for model versioning process, where the generated code is expected to be complete with
in the evaluated tools. The first, locking, does not scale to large no manual editing required. The generators are designed to produce
collaborative projects. The second, leaving versioning completely to code that can be integrated in a larger project.
Magic Draw can generate code in several languages (Java, C++, scenario. We note that this type of synchronization functionality
C#). In most cases, code generated from models will be skeletons focuses on models and code created in a single modeling tool. In
and thus will be edited by developers to implement complete func- projects where multiple modeling tools are used, more challenges
tionalities. There is also support for reverse engineering; models related to the synchronization of the different models can be ex-
can be derived from code. Forward and reverse code engineering pected. This is mainly due to the limitation of impact analysis to
is managed using Code engineering sets. These sets contain model assess only the impact of changes in models to models created in
elements for which code is generated and conversely files from the same tool.
which code is reversed to models. In addition, relationships be-
tween model components can be defined. These can be visualized 4.3.3 Testing. To test models, BridgePoint provides a verifier.
in different ways to show the impact of changes on the remaining This functionality forms the “eXecutable” part of xtUML. The ver-
model artefacts in the project. ifier can be used to test models without the need to regenerate
Papyrus supports code generation from UML models through source code. It executes the model itself and supports placement
plug-ins. There are plug-ins available for the generation of C++ of breakpoints and inspection of variable values during the sim-
and Java code from UML models, but it is also possible to create ulations. This can be useful for manual testing, but less so in CI
custom code generators for other languages. Reverse engineering settings, where automated testing is preferred.
is supported as part of the Papyrus Software Designer tooling, In Enterprise Architect, models can be simulated and test
using it, class diagrams can be generated from Java classes and scripts can be defined to automatically test model elements. In these
packages. Using the Papyrus Software Designer plug-in, models scripts, unit tests for Java (JUnit) or .NET (NUnit) can be called. A
and generated code can be synchronized. Changes to the code are skeleton for these unit tests can automatically be generated from
then propagated back to the model and changes in the model are class diagrams. By default, the tool supports the validation of
incrementally applied to the code. models with respect to the UML syntax, but custom rules can be
Rhapsody can generate code in C, C++, Java and, using a spe- added. Furthermore, validation and test scripts can be used to
cific Rhapsody Developer version, also for Ada. This is done automate testing of models, whereas the simulation functionality
incrementally, i.e., only new code is generated for modified model is mostly meant for debugging.
elements. The generated code can be modified and changes are LabVIEW includes a framework for unit testing. Test cases can
propagated back to the model. It is also possible to specify code be defined in the tool itself by defining input values and expected
that should not be included in this round-trip, which could be use- output values for a specific unit under test. The tests can be executed
ful for implementation-specific code that is not to be reused in in isolation or in a test suite. The tool includes a functionality
other versions of a product. To see the impact of a change on the to track tests and the code it covers, automatically providing the
other artefacts in the models, Rhapsody supports automated im- developer with code coverage information. In addition to this,
pact analysis. The user configures the analysis by defining, among models can be validated using static code analysis rules, which can
other things, the types of links to follow and their depth. Given the be customized for particular purposes.
result of an impact analysis, it is up to the developer to manually Integrity Modeler contains a framework for automated test-
co-evolve the impacted artefacts. ing. In it, test cases can be defined, triggered and their results
The generation of C and C++ code from Simulink models is viewed. The test cases can also be grouped in sessions, allowing
supported by additional tools that can be integrated in Simulink, their execution to be automated.
such as Embedded Coder and Simulink Coder. Generated code is a In Magic Draw, models can be validated with respect to pre-
complete program, not just skeleton code. Modifying the generated defined constraints or custom created constraints expressed in the
code can be done at the level of the code generators, which can be Object Constraint Language (OCL). If the validation logic cannot be
configured to replace code by custom snippets. This also means expressed in OCL, boolean constraints can be defined in Java. Ad-
that the process of code generation is one-way; there is no support ditionally, unit tests can be defined to verify models or integrations.
for propagating manual changes in the generated code back to JUnit is used to express test cases that can be executed using the
the model. Simulink also contains a facility for automated impact build-in test framework. The framework also provides functionality
analysis that can predict impacted elements in anticipation of a for checking the created program for memory leaks.
particular change. Papyrus models can be validated with respect to predefined
soundness constraints. Custom constraints can be defined in OCL.
Summary. The basic functionality of generating skeleton code Validations can be performed on an entire model as well as on
from models is present in each of the considered tools. The detail of parts of a model. Warnings or errors are shown in the models
the created models dictates whether the entire application or only themselves after a validation. This validation is a static check, but
skeleton code can be generated. This distinction usually influences UML models can also be executed, using the execution engine of
the functionality regarding synchronization of models and code too. the Moka module. This can be used to manually test models, but
This aspect is usually better supported in tools that just produce there is also support for automatic testing. The Papyrus Testing
skeleton code than in tools that produce complete code and where Framework supports the automatic generation of unit tests from
thus the generated code does not require manual editing. We have UML diagrams. The unit tests can be automatically tested using the
seen that some tools contain functionality to assess the impact JUnit framework.
of changes at model level, but that the implementations still rely Before code is generated in Rhapsody, a model-checker can be
mostly on manual actions, which is not ideal in an automated build run to validate the model with respect to predefined and custom
defined rules. Such custom rules have to be written in Java and applicable in fewer cases than the generally applicable batch mode
can be used to check both the structural and the behavioral aspects construct.
of the model. Included in the tool is also a functionality to simu- For Papyrus, code generation and model testing functionalities
late models (or animate as is the used terminology for this tool). are packaged in Eclipse plug-ins. These are executable from the
In addition, the tool can be integrated with other IBM Rational command line, which can be leveraged to include Papyrus in a CI
tools for testing (Test RealTime) and quality assessment (Quality pipeline, for example by calling these plug-ins from scripts man-
Manager). Furthermore, the tool includes a framework for the aged by a CI server such as Jenkins. Creating a CI pipeline for
automatic generation of test cases. conventional Eclipse projects is a common practice, so it is not
Similar to code generation, there exist additional tools for the expected that these particular tools would yield new problems.
validation and testing of Simulink models. Simulink Test is a tool Rhapsody offers command line interfaces for code generation
that supports creation and execution of test cases for models. Test and the DiffMerge tool. Using these commands, code can be
cases can be defined to verify the models with respect to functional generated for specific components or for a project containing a
constraints. It also provides an overview of failed and succeeded number of components. This allows for these tasks to be integrated
test cases, similar to the dashboard of CI tools. in a CI pipeline where only models are checked in to the version
control system and the application is generated.
Summary. Most tools contain a unit testing mechanism. Some It is possible to create a CI pipeline using Jenkins to automat-
implement their own and some use existing frameworks such as ically execute builds and tests in Simulink. Furthermore, the CI
JUnit. Most tools also include model validation functionality, a tool can be configured to report on the success or failure of the
check of the well-formedness of the models with respect to the automated tests. Such a process can be created using MATLAB, Git
metamodel. Additionally, some tools are capable of simulating and Jenkins [8].
models, which is primarily useful for manual debugging. Next, we
look at ways to automate builds and tests in the considered tools.
Summary. With some effort, each modeling tool can be included
4.3.4 Automation. The BridgePoint editor is based on Eclipse in a CI pipeline. We have seen some examples of ready-made
and its model compilers are implemented as Eclipse plug-ins. It is pipelines including some of the discussed tools. As long as the
possible to run these from the command line and thus incorporate different approaches to automation can still be executed from a
them as a build step in a CI pipeline. Similarly, the testing function- pipeline, there should be no impediments regarding combining
ality incorporated in the verifier can be included in an automated automation for multiple modeling tools in one pipeline.
process.
Enterprise Architect offers the possibility to create analyzer 4.3.5 Evaluation Summary. Table 3 summarizes the evaluations
scripts, these can be used to automate builds, tests, and other func- by scoring the different aspects in each tool as mature, standard,
tionalities. The scripts can be created in the tool itself and allow for or immature. We define the thresholds for these scores, based on
execution of the builds and tests from the command line. The scripts the identified aspects in Section 3, as follows. For integration, an
can also be used to specify an output file to contain a generated immature level of support is considered an approach that does not
report on the test results. Furthermore, they can be used to execute allow versioning of models, but e.g. utilizes pessimistic locking.
the models and deploy the project, but that is out of the scope of A standard support would be one where models can be merged.
our definition of CI. A mature level of support is considered when the tool also sup-
Automation for CI can easily be achieved in Integrity Modeler ports the visualization and resolution of conflicts. For building, an
using a Jenkins plug-in.3 The plug-in can detect changes in the built- immature level of support would be one where no code genera-
in repository and can be configured to execute builds after such tion is possible. Code generation is considered standard support,
a detection. Furthermore, it can retrieve the results of automated while mature support includes back-propagation of code changes
tests, executed after the build. The availability of the Jenkins plug-in to models. For testing, immature level of support only includes
signals a higher level of maturity with respect to CI processes than syntactical validation of models. Standard support involves also
seen in other tools. verification of models using model testing. Support for testing is
For LabVIEW, command line interfaces are available as open- considered mature when it includes unit tests for code and models.
source.4 These allow the builds and tests to be executed by a CI For automation, tools are considered to provide immature support
server, e.g. Jenkins. The test reports created by the tool can be if they do not feature explicit hooks to incorporate them in a CI
stored as HTML files and as such be shown in Jenkins [9]. pipeline. A standard level of support includes the possibility to run
Magic Draw supports extensibility by custom add-ins through the tool in batch mode and perform some actions. We require the
its Open API. This API also allows the tool to be run in batch mode. presence of a command-line API to grant mature level of support
This allows command line access to code generation and unit test for automation.
execution, which makes it suitable to be used in a CI pipeline. Alter-
natively, Magic Draw can also be integrated in other applications
using its OSGi interfaces. Since this construct is Java-based, it is
5 DISCUSSION
We have summarized the answers to our research questions by
3 https://wiki.jenkins.io/display/JENKINS/PTC+Integrity+Plugin Last access: June 4, listing relevant features for a modeling tool to be able to support
2018 CI practices in Table 1 and by summarizing the extent to which
4 https://github.com/JamesMc86/LabVIEW-CLI, retrieved: June 1, 2018 these features are present in current modeling tools in Table 3.
Table 3: Aspects as supported by tools, scored by -, ◦, or +, de- Similarly, the selection of the tools contains a bias towards UML
picting immature, standard, or mature support respectively. and SysML. Other languages or modeling paradigms may yield
different impediments.
Tools Other threats are related to our chosen research methodology.
The evaluations are based on publicly available documentation
Magic Draw
BridgePoint
Enterprise
and research papers. This means that we did not experiment with
Rhapsody
Architect
Simulink
Integrity
LabView
Modeler
Papyrus
the tools themselves to assess the aspects. The advantage of this
approach is that we avoid defining a scenario to evaluate the tools,
which may not fit all tools or may accidentally favor some of them.
Integration - - ◦ + + + + + On the other hand, the inherent threat of this approach is that
Aspects
Building ◦ + + ◦ + + + ◦ it does not highlight possible issues only visible when using the
Testing ◦ + ◦ + ◦ + + ◦ evaluated tools in practice.
Automation - ◦ + + ◦ ◦ + +
7 CONCLUSIONS AND FUTURE WORK
In this paper, we identified relevant aspects of modeling tools to
support CI practices. We then evaluated eight modeling tools and
Regarding integration, most of the considered tools provide sup-
assessed their levels of support for each of the aspects. In the
port for differencing and merging at model level. They provide
evaluated tools, we have seen different maturity levels of support
representations in various formats of model differences and allow
for the considered aspects. Overall, we found some challenges,
developers to resolve merge conflicts at model level. The storage
but no insurmountable impediments to introducing CI practices in
of models and change history is usually managed by a VCS such
MBD.
as SVN or Git. For building, all tools provide some form of code
The next step of our research consists in a set of interviews
generation. Nonetheless, there is a great variability in the maturity
with MBD practitioners working at different MBD maturity levels
of what the environments can do after the code is generated. Some
and in different companies. Practitioners will be asked to share
tools automatically keep code and models synchronized whereas
their views on introducing CI practices in MBD and their views on
in others code generation is a one-way operation. The most chal-
the impediments in their context. These interviews may uncover
lenging aspect seems to be the synchronization of different models,
hidden technical aspects that did not arise in this work, or bring up
for which most of the tools include some level of change impact
other, non-technical impediments, such as those briefly mentioned
analysis support. Another challenge is the synchronization of mod-
in Section 5.
els when multiple modeling tools are used in the same software
project and combined in the same pipeline. Testing is supported
by each of the tools, but with different maturity levels. Finally, it
8 ACKNOWLEDGEMENTS
is possible to automate at least parts of the build and testing pro- The authors would like to thank the industrial partners for their
cesses for each of the tools. For some of the modeling tools such input in discussions about this work. This work is part of a project
automated approaches are available, whereas for others it would supported by Software Center.5
require custom configuration.
From the evaluations of the modeling tools, we did not find any REFERENCES
theoretical obstacle in introducing CI practices in MBD. Some tools [1] Hessa Alfraihi and Kevin Lano. 2017. The Integration of Agile Development and
Model Driven Development - A Systematic Literature Review. In Proceedings
already have fairly good support for CI, and by cherry picking of the 5th International Conference on Model-Driven Engineering and Software
features from other tools, they could be even more suitable for Development (MODELSWARD 2017). SCITEPRESS, 451–458.
CI practices. In fact, in Section 4.3.4 we have mentioned some [2] Hessa Alfraihi and Kevin Lano. 2017. A Process for Integrating Agile Software
Development and Model-Driven Development. In Proceedings of MODELS 2017
applications of CI in each of the tools Integrity Modeler, Lab- Satellite Event: FlexMDE. 412–417.
VIEW, and Simulink. On the other hand, we foresee challenges [3] Kerstin Altmanninger, Martina Seidl, and Manuel Wimmer. 2009. A Survey on
Model Versioning Approaches. International Journal of Web Information Systems
in model synchronization and automation in projects involving (IJWIS) 5, 3 (2009), 271–304.
multiple modeling tools. [4] Scott W Ambler. 2003. Agile Model Driven Development is Good Enough. IEEE
Software 20, 5 (2003), 71–73.
[5] Paul Baker, Shiou Loh, and Frank Weil. 2005. Model-Driven Engineering in a
6 THREATS TO VALIDITY Large Industrial Context—Motorola Case Study. In LNCS 3713. Springer, 476–491.
In drafting both the list of relevant aspects and the list of considered [6] Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward Cun-
ningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron
tools there is an amount of subjectivity and possible bias. Tools Jeffries, and others. 2001. Manifesto for Agile Software Development. (2001).
and aspects may be omitted or be listed but less relevant. Incom- http://agilemanifesto.org
[7] Francis Bordeleau, Grischa Liebel, Alexander Raschke, Gerald Stieglbauer, and
pleteness of the list of aspects was a threat to the validity of this Matthias Tichy. 2017. Challenges and Research Directions for Successfully Ap-
work, since we aim to find impeding aspects. If crucial aspects plying MBE Tools in Practice. In Proceedings of the 20th International Conference
were not included, we could not find the corresponding problems on Model Driven Engineering Languages and Systems (MODELS). 338–343.
[8] Andy Campbell. 2015. The Other Kind of Continuous Integra-
in the evaluated tools. To limit this risk, we gathered core rele- tion. (2015). https://blogs.mathworks.com/developer/2015/01/20/
vant aspects by investigating existing literature on the topics of the-other-kind-of-continuous-integration Retrieved: 2018-05-14.
CI and MBD; furthermore, the lists were informally validated by
two practitioners from industry with experience in MBD and CI. 5 www.software-center.se
[9] Fredrik Edling. 2013. Using LabVIEW in a Continuous Integration Environment. [23] Erkuden Rios, Teodora Bozheva, Aitor Bediaga, and Nathalie Guilloreau. 2006.
(2013). ftp://ftp.ni.com/pub/branches/northern region/nidays2013/presentations/ MDD Maturity Model: A Roadmap for Introducing Model-Driven Develop-
sw cag using lv in cont integr environment.pdf Retrieved: 2018-06-01. ment. In Proceedings of the European Conference on Model Driven Architecture-
[10] Jean-Marie Favre. 2004. Towards a Basic Theory to Model Model Driven En- Foundations and Applications. Springer, 78–89.
gineering. In 3rd Workshop in Software Model Engineering, WiSME. Citeseer, [24] Douglas C Schmidt. 2006. Model-Driven Engineering. IEEE Computer 39, 2
262–271. (2006), 25.
[11] Martin Fowler. 2006. Continuous Integration. (2006). https://martinfowler.com/ [25] Bran Selic. 2003. The Pragmatics of Model-Driven Development. IEEE software
articles/continuousIntegration.html 20, 5 (2003), 19–25.
[12] Jokin Garcia. 2018. Continuous Model-Driven Engineering. (2018). https: [26] Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous
//modeling-languages.com/continuous-model-driven-engineering/ Retrieved: Integration, Delivery and Deployment: a Systematic Review on Approaches,
2018-05-14. Tools, Challenges and Practices. IEEE Access 5 (2017), 3909–3943.
[13] Jokin Garcia and Jordi Cabot. 2018. Stepwise Adoption of Continuous Delivery [27] Daniel Ståhl and Jan Bosch. 2013. Experienced Benefits of Continuous Integration
in Model-Driven Engineering – Extended Abstract. DEVOPS (2018). in Industry Software Product Development: A Case Study. In The 12th IASTED
[14] Vicente Garcı́a-Dı́az, Jordán Pascual Espada, Edward Rolando Núnez-Valdéz, G International Conference on Software Engineering (Innsbruck, Austria, 2013). 736–
Pelayo, B Cristina Bustelo, and Juan Manuel Cueva Lovelle. 2016. Combining the 743.
Continuous Integration Practice and the Model-Driven Engineering Approach. [28] Daniel Ståhl, Torvald Mårtensson, and Jan Bosch. 2017. Continuous Practices
Computing and Informatics 35, 2 (2016), 299–337. and DevOps: Beyond the Buzz, What Does It All Mean?. In Software Engineering
[15] Sebastian Hansson, Yu Zhao, and Håkan Burden. How MAD are we? Empirical and Advanced Applications (SEAA), 2017 43rd Euromicro Conference on. IEEE,
Evidence for Model-driven Agile Development. In Proceedings of XM 2014, 3rd 440–448.
Extreme Modeling Workshop, Vol. 1239. 2–11. [29] Stavros Stavru, Iva Krasteva, and Sylvia Ilieva. 2013. Challenges of Model-driven
[16] John Hutchinson, Jon Whittle, Mark Rouncefield, and Steinar Kristoffersen. 2011. Modernization-An Agile Perspective.. In MODELSWARD. 219–230.
Empirical Assessment of MDE in Industry. In Proceedings of the 33rd International [30] Henrik Steudel, Regina Hebig, and Holger Giese. 2012. A Build Server for
Conference on Software Engineering (ICSE). IEEE, 471–480. Model-Driven Engineering. In Proceedings of the 6th International Workshop on
[17] Melanie Langermeier, Christian Saad, and Bernhard Bauer. 2014. Adaptive Multi-Paradigm Modeling. ACM, 67–72.
Approach for Impact Analysis in Enterprise Architectures. In International Sym- [31] Nenad Ukić, Pál L Pályi, Marijan Zemljić, Domonkos Asztalos, and Ivan Markota.
posium on Business Modeling and Software Design. Springer, 22–42. 2011. Evaluation of Bridgepoint Model-Driven Development Tool in Distributed
[18] Grischa Liebel, Nadja Marko, Matthias Tichy, Andrea Leitner, and Jörgen Hans- Environment. In Workshop on Information and Communication Technologies
son. 2016. Model-Based Engineering in the Embedded Systems Domain: an conjoint with 19th International Conference on Software, Telecommunications and
Industrial Survey on the State-of-Practice. Software & Systems Modeling 17, 1 Computer Networks, SoftCOM 2011.
(2016), 91–113. [32] Ragnhild Van Der Straeten, Tom Mens, and Stefan Van Baelen. 2008. Challenges
[19] Torvald Mårtensson, Daniel Ståhl, and Jan Bosch. 2017. Continuous Integration in Model-Driven Software Engineering. In International Conference on Model
Impediments in Large-Scale Industry Projects. In Proceedings of the 2017 IEEE Driven Engineering Languages and Systems. Springer, 35–47.
International Conference on Software Architecture (ICSA). IEEE, 169–178. [33] Jon Whittle, John Hutchinson, Mark Rouncefield, Håkan Burden, and Rogardt
[20] Ade Miller. 2008. A Hundred Days of Continuous Integration. In Agile. IEEE, Heldal. 2013. Industrial Adoption of Model-Driven Engineering: Are the Tools
289–293. Really the Problem?. In International Conference on Model Driven Engineering
[21] Parastoo Mohagheghi, Wasif Gilani, Alin Stefanescu, and Miguel A. Fernandez. Languages and Systems. Springer, 1–17.
2013. An Empirical Study of the State of the Practice and Acceptance of Model- [34] Yuefeng Zhang and Shailesh Patel. 2011. Agile Model-Driven Development in
Driven Engineering in Four Industrial Cases. Empirical Software Engineering 18, Practice. IEEE software 28, 2 (2011), 84–91.
1 (01 Feb 2013), 89–116.
[22] David Norfolk. 2015. PTC Integrity Modeler — a Standards-Based Tool for Systems
and Software Engineering. Technical Report.