Continuous integration support in modeling tools Robbert Jongeling, Jan Carlson, Antonio Cicchetti, Federico Ciccozzi Mälardalen University, Västerås, Sweden Email: {robbert.jongeling, jan.carlson, antonio.cicchetti, federico.ciccozzi}@mdh.se ABSTRACT practices regarding the frequency and level (e.g. the entire system, Continuous Integration (CI) and Model-Based Development (MBD) a sub-system or an own branch) of software integrations have been have both been hailed as practices that improve the productivity of developed [19]. The terms Continuous Integration, Continuous software development. Their combination has the potential to boost Deployment and Continuous Delivery are sometimes used inter- productivity even more. The goal of our research is to identify im- changeably [26]. An unclear definition and interchangeable use of pediments to realizing this combination in industrial collaborative the terms may lead to their devaluation [28]. Therefore, we refer to modeling practices. In this paper, we examine certain specific fea- the definition of Continuous Integration, given by Fowler [11], as tures of modeling tools that, due to their immaturity, may represent follows: impediments to combining CI and MBD. To this end, we identify Definition 1. Continuous Integration is a collaborative develop- features of modeling tools that are relevant to enabling CI practices ment practice where software engineers frequently, at least daily, in MBD processes and we review modeling tools with respect to integrate their work into a shared repository. After each integration, their level of support for each of these features. an automatic build is performed. On successful build, automated tests are performed. 1 INTRODUCTION In MBD, this integrated work consists of models and other mod- In this work we couple two concepts: Continuous Integration (CI) eling artefacts in addition to code. This may pose additional chal- and Model-Based Development (MBD). CI refers to a subset of the lenges, such as differencing on model level, that are not encountered Agile development practices as described by Martin Fowler [11]. when applying CI practices in conventional software development Several empirical evaluations have shown the positive effects of CI projects. on productivity in industrial software development projects [20, 27]. The MBD paradigm holds the promise of increased productivity of 1.2 Model-Based Development development teams by raising the level of abstraction from code We use the term Model-Based Development (MBD) to denote mod- to models [24]. The benefits of MBD on productivity have been eling practices in which models are used to capture functionality empirically assessed in industrial settings too [5, 16, 21]. and possibly to generate code. Rios et al. distinguish five maturity We hypothesize that CI practices can further improve the produc- levels of modeling practices [23]. The levels range from ad-hoc to tivity of MBD. In our research, we aim at identifying impediments to ultimate and describe immature to complete modeling practices. realizing these practices in industry. Eventually, although not in the Since our goal is to introduce CI practices in MBD, it does not make scope of this paper, we aim at resolving them, thereby contributing sense to consider the first level, which describes immature modeling to the maturity of collaborative modeling practices. practices where models are only used by individuals for e.g. design In this work, we focus on technical challenges towards introduc- or documentation. Instead, we are interested the more advanced ing CI practices in MBD. We examine modeling tools to identify levels of modeling practices where models are used by multiple particular features that are commonly underdeveloped and thereby team members and the eventual code or application derived from may represent potential impediments to combining CI and MBD. the models must be consistent with these models. In particular, we answer the following research questions: (1) What are relevant features for a modeling tool to be able 2 RELATED WORK to support CI practices? Since the publication of the agile manifesto in 2001, research on (2) To what extent are these features provided in current mod- combining Agile and modeling practices [4], has been performed. eling tools? Evidence-based software engineering studies of the field have shown In the remainder of this introduction, the concepts of CI and that Agile modeling is not a mature field yet [1, 15]. In particular, MBD are described in more detail. The rest of the paper is organized these studies identify the need for more reports on Agile modeling as follows. Section 2 outlines related reviews of modeling tools. practices in industry. The first research question is answered in Section 3. In Section 4, Although Agile modeling is a topic treated in several research the second research question is answered. The results are discussed articles and CI is named as a needed practice in modeling [2], we in Section 5. Threats to the validity of our work are outlined in have only found few that go into the details of the Agile practice Section 6 and the conclusions are provided in Section 7. of CI in combination with MBD. Garcı́a-Dı́az et al. identify four problems in applying CI practices in an MBD project [14]. Among 1.1 Continuous Integration these problems, the two most interesting in relation to CI (as we CI is one of the twelve Extreme Programming (XP) practices [11]. will see later in Section 3) are version control systems for models In turn, XP is one of the elements of the software development and incremental code generation. Additionally, they stress the im- concepts published in the Agile manifesto [6]. Since then, various portance of uniform, user-friendly interfaces and the variability of technologies in different phases of the pipeline. They build and 3.1 Core CI Aspects evaluate a prototype solution to resolve these identified problems In their literature study, Shahin et al. provide an overview of in an MBD project. This solution focuses on modeling approaches types of tools used to form a pipeline for Continuous Deployment where models are used to generate 100% of the code for an applica- (CD) [26]. The categories are: Version Control System, Code Manage- tion, whereas we consider also modeling approaches where only ment and Analysis, Build System, CI Server, Testing, Configuration parts of the code are generated. and Provisioning, and CD Server. They note that an implementation Some work has been done towards a build server for MBD [30]. of CD does not necessarily include all categories. Furthermore, the The authors identify the need to support verification and validation set of CI practices can be considered a part of that CD pipeline, of models. Furthermore, they argue the need for build tooling to where the latter two tool categories (i.e. Configuration and Provi- support a mix of automatic and manual actions. sioning and CD Server) are excluded. Nevertheless, the CD pipeline Recently, early work towards resolving impediments to combin- is a good starting point to find out relevant features for combining ing Continuous Delivery and MBD was presented [12, 13]. The CI and MBD. authors identify the main technical impediment to be the model- We now elaborate on each of the proposed categories and their awareness of the integration server. Furthermore, they remark that relevance for CI practices in MBD. A Version Control System (VCS) for all typical MBD activities, tools that can be included in a pipeline is used to manage different versions of developed artefacts. The are available. Our approach looks at this from a different perspec- artefacts are typically stored in a shared repository, sometimes tive; we explore modeling tools and then consider the possibilities allowing developers to copy it and work on their own instance, or to introduce CI practices. branch. Integration of code is done into this shared repository. Code Naturally, there are aspects of MBD in general, not just the com- management and analysis techniques such as static code (or model) bination of CI and MBD, that are relevant to its adoption in industry. analysis might be employed to improve the quality of artefacts in a The impediments may be technical or be related to organizational CI process. They are however themselves not closely related to the factors of the software development process. Tooling is one of three main activities in the definition of CI: integrating, building the aspects preventing a more wide-spread adoption of MBD in and testing. Therefore, we do not include the Code Management and industry; another is the lack of clear processes to support devel- Analysis category. A build system typically combines artefacts to opment [33]. There are numerous organizational challenges that, create executables. In the case of MBD, this could mean generating if not properly tackled, hinder agility in MBD [29]. Other impedi- code from models but also keeping the consistency of several related ments to the adoption of MBD that are highlighted in literature are models. CI servers do not perform builds themselves but rather the lack of tool interoperability [18], and the steep learning curve execute builds and automated tests in other tools, the results are for developers [5, 29]. Furthermore, impediments might lie in devel- combined in a status overview [11]. These automated tests are opment processes of companies and the required time and money important to indicate the quality of integrations. In that way, they investments to change these. Technical challenges include poor contribute to increasing the predictability of the amount of work scalability of the modeling practice in general in large industrial left, one of the purposes of CI. applications of MBD [5]. Related to these are automated test gener- ation and performance of generated code. Finally, Selic mentions 3.2 MBD Aspects Related to the Core challenges regarding the integration with legacy systems, trace- Continuous Integration CI Aspects ability of generated code and the ability to execute models [25]. Nevertheless, we limit the scope of this paper to those aspects We differentiate between three types of projects to which CI can be relevant for introducing CI practices in MBD. applied. The first type is traditional software development, no mod- In megamodeling, a model is created to describe the relationships els are used at all. The second type is very mature MBD, where all between all concepts in a modeling project [10]. Since part of the code is generated from models and no manual coding is performed. work to enable CI practices is to chart the artefacts in a project and The third type concerns less mature kinds of MBD, where code is how they are related, a CI pipeline could be seen as a megamodel. partially generated and then manually extended to form a complete In this paper, we do not further explore this relation and instead application. We consider the latter two types in this paper. The focus on existing modeling tools that are used in current practice. inclusion of modeling artefacts in the CI process requires that some parts of integration, testing and building are handled differently. In this section, we identify the specific aspects of MBD that constitute these differences. 3.2.1 Integration. We consider the integration of changes to individual artefacts into the repository and focus on the aspects 3 IDENTIFYING RELEVANT ASPECTS of model differencing and model merging to facilitate this inte- In this section, we identify aspects of modeling tools that are rele- gration. Integration is thus considered an activity localized to the vant for enabling CI practices in MBD. We first identify core aspects directly changed artefacts. If multiple artefacts need to be changed of CI based on our definition and existing literature. Then, we list to maintain consistency of the models, this is considered as the particular aspects that realize these general CI aspects in the MBD responsibility of the developer. domain, based on existing MBD literature. We submitted the identi- In order to integrate models in a shared repository, there must fied aspects for review to two industrial practitioners of MBD. The be a way to control their different versions. Zhang and Patel [34] resulting aspects are summarized in Table 1. refer to CI in MBD as “Continuous Modeling.” They identify the need to merge frequently, but also note that merge tooling cannot analogous to traditional software development in which e.g. non- handle many simultaneous changes. Merging of models is also compiling or incorrect code is integrated. In such cases, the builds identified as an important aspect to the adoption of modeling tools or tests are expected to fail. in general [7, 29]. Alternatively, pessimistic locking is used to avoid merge conflicts by allowing only one developer at a time to make changes to a model or part of a model [3]. 3.2.4 Model-Awareness. In some work about combining CI and MBD, authors have argued the need for more model-awareness in tasks related to CI. In case of the build tooling, it is argued that 3.2.2 Building. There is no direct MBD equivalent of a build sys- manual actions that are typically performed in MBD should be tem for conventional programming languages. A build system for taken into account and that testing should focus on validation and models requires more steps than a build system for code, since code verification of models [30]. Others argue that the entire pipeline needs first to be generated from the models and possibly altered or should be model-aware, such that dependencies between artefacts completed by developers. Given the scope of our research, we adapt and between jobs in the pipeline that would be lost in textual the previously identified aspect of a build server to include more representations can be discovered [13]. Garcı́a-Dı́az et al. also note model-specific actions. Automated code generation is a central part lack of model-awareness, particularly in version control systems of continuous integration in a modeling context [34]. Furthermore, and code generation [14]. We do not add an aspect model-awareness it is argued that code generators should work incrementally, i.e., to our list but when discussing the other aspects in modeling tools, that code should be generated only for parts of the model that have we do take into account to what extent their implementation is changed [14]. But the key elements of building in MBD are the model-aware. ability to generate code and the ability to synchronize models and The need for model-awareness may also refer to the need to syn- code. Therefore, we add only code generation and model discovery chronize several models when one of them is changed. In our case, as relevant aspects. this model synchronization is limited to the consistency of models For the different types of CI projects, these aspects can have and code, and between several models in a single project. In other different meanings. In projects with complete code generation, this cases, the term co-evolution is used to refer to similar activities that generation is a task for the build server and is not performed locally also include the synchronization of models and metamodels, or the by developers. When code is only partially generated, this can be synchronization of models and model transformations. Since the done both locally and on the build server. In case of complete code most used metamodels in the considered tools are UML and SysML, generation, model discovery is irrelevant, it will never be done since they and the model-to-text transformations (code generators) are the code is never manually edited. Conversely, in a partial code typically not changed during a project. We therefore refer to this generation scenario, model discovery can be needed both locally MBD aspect as “model synchronization.” and remotely. In the simplest form, a developer locally makes Extensive support for activities related to model synchroniza- changes to a piece of generated code and immediately updates the tion, such as automatically handling inconsistencies, is required corresponding model too. In a more complex scenario, a developer in modeling tools [32]. Model synchronization is also related to could only change the code, integrate it and expect the build server code generation and model discovery, i.e., the automatic creation to propagate her changes to the model. of a model from code. Since the generated code and models can become inconsistent after changes to either. Modeling tools can 3.2.3 Testing. We distinguish three components of testing in support this synchronization, e.g. by providing automated impact MBD: validating the model with respect to syntax, verifying the analysis for changed artefacts, but the process cannot always be model with respect to predefined requirements and verifying mod- automated. Therefore, manual actions could be required during els after integration (integration testing). The aforementioned con- model synchronization, this step is unsuitable for inclusion in au- tinuous modeling practices specify unit and integration testing as tomated builds. Since we envision a CI pipeline for models that is important practices to build confidence in the product [34]. Veri- automated similarly to that for traditional software development, fication and validation of models are identified as crucial features we consider model synchronization to be a task performed locally of a build server for MBD [30]. In both cases, “testing” specifically by a developer. Consequently, the CI server or build server is not refers to testing the correctness of the created models, it is assumed concerned with tasks such as propagating changes to other arte- that correct models yield correct code and thus correct applica- facts. The identified need for support is still relevant, since the tions. We therefore add model validation and verification, as well developer should be supported in her local work. as integration testing, as sub-categories of the testing aspect. Again, some distinctions can be made in testing between the different types of CI projects. In case of complete code generation, 3.2.5 Automation. Interoperability of a modeling tool with other testing the models and the generated code should yield the same re- tools is an important aspect to consider too [34]. To assess their sults. In partial code generation projects, code is the predominantly suitability to cooperate with CI servers, we look at possibilities to tested artefact. In both cases, validation of models with respect to run modeling tools in batch mode or call their functionality from syntax is an implicit part of the code generation process, which the command line. If this functionality exists, it can be used to will not yield correct output for invalid models. This validation can create a script that automates part of the CI pipeline, including also be a local action, but this is not required for the CI process. building and testing. Such automation is of crucial importance to It does not prevent the integration of invalid or incorrect models, the adoption of CI practices in industry. 3.2.6 Summary. The aspects identified in this section are sum- UML and SysML, among the most used modeling languages in in- marized in Table 1. It contains primary aspects (in bold) and sec- dustry [18]. Exceptions to this are BridgePoint, which supports ondary, more specific aspects. In Section 4, we evaluate a set of xtUML (an executable dialect of UML), LabVIEW, which supports modeling tools with respect to these primary aspects based on their their “G” graphical modeling language, and Simulink, which sup- support of the secondary aspects. Notably, not all CI aspects are ports modeling in the Simulink language. Most of the tools thus directly mapped to a single MBD aspect. Rather, the MBD aspects support general purpose modeling languages. The most advanced are specific to their domain and target a more specific function- modeling practice includes the creation of custom Domain-Specific ality than the general CI aspects. In the table, the citations refer Languages (DSLs). Tools used for that purpose are further away to literature sources used to identify the relevance of the related from the state of practice at our industrial partners and therefore aspects. not included in this evaluation. An overview of the considered tools is shown in Table 2. Table 1: Identified relevant aspects of CI in MBD. Table 2: Evaluated tools in this review. CI MBD Supported Integration [26] [7, 29, 34] Modeling Model Differencing Tool Vendor Languages Model Merging Building [26] BridgePoint OneFact xtUML Code Generation [14, 34] Enterprise Architect Sparx Systems UML + SysML Model Discovery [32] Integrity Modeler PTC UML + SysML Aspects Model Synchronization [12, 32] LabVIEW National Instruments G Testing [26] Magic Draw No Magic UML + SysML Model V&V [30] Papyrus Eclipse UML + SysML Integration Testing [34] Rhapsody IBM UML + SysML Automation [26] [12] Simulink MathWorks Simulink Building Testing 4.2 Other Tools In addition to the modeling tools, a CI pipeline typically also in- 4 SUPPORTED ASPECTS volves Version Control Systems (VCSs) and CI servers. There exist In this section, we introduce the evaluated modeling tools and numerous open-source and commercial CI servers, such as Jenk- discuss for each tool how it implements support for the primary ins, Travis, and TeamCity. Some of these allow for a completely aspects in Table 1. We discuss the tools with respect to their support custom defined pipeline whereas other tools provide users with a for the relevant aspects of CI in MBD as discussed in Section 3. choice between predefined pipelines for some programming lan- Note that the selection of modeling tool(s) depends on more than guages. Since we aim at using these tools in MBD processes, we are just the ability to use it in a CI process applied to an MBD project. particularly interested in those CI servers that allow the definition Conversely, a CI process in MBD depends on more than just the of a custom pipeline. Therefore, we will mainly refer to Jenkins in used modeling tool(s), such as the maturity level of the modeling the remainder of this section. practices. The goal of our work is not to identify the best modeling tool for CI, but rather to investigate what impediments in applying 4.3 Tool Evaluations CI to MBD projects exist. We evaluated how the selected tools support the primary aspects depicted in Table 1. The evaluations are based on publicly available 4.1 Modeling Tools documentation and research papers about the tools. The results of The selection of modeling tools was based on their use in industry the evaluation are summarized at the end of this section in Table 3. and on inputs from our industrial partners. We included the four most-used1 tools in industry as reported by practitioners [18]: Mat- 4.3.1 Integration. In BridgePoint, version control is difficult lab Simulink, Sparx Systems Enterprise Architect, IBM Ra- to achieve [31]. Automated merging is not supported but the tool tional Rhapsody2 , and National Instruments LabVIEW. After does show a visual difference between two versions of a model. discussions with our industrial partners, this list was supplemented This is not necessarily an impediment to introducing CI practices, with four additional tools that are most relevant to their daily but in practice it may discourage developers to integrate frequently work: NoMagic Magic Draw, PTC Integrity Modeler, OneFact if every integration potentially requires a large manual effort. BridgePoint, and Eclipse Papyrus. Most of these tools support Enterprise Architect (EA) has no integrated support for model versioning but relies on pessimistic locking of packages (parts of 1 Excluding Eclipse-based tools and in-house tools, since they cannot be specified to a models). This system grants user exclusive editing rights on a particular tool. They are reported as second and fourth most used respectively [18]. 2 In [18] the reported tool is Rational Modeler, but we include Rational Rhapsody, package, thus preventing conflicts due to simultaneous changes. which can be seen as its successor. The tool does support the integration of several third-party version control systems, which can be used to store and manage the history a VCS, is not feasible because the VCS is typically not model-aware, of EA models. Additionally, LemonTree is a third-party project which is required for merging at model level. Furthermore, line that supports optimistic locking, three-way merging and branching based differencing of the XML representations is not appropriate for EA models. for models [3]. The third and most feasible versioning approach Integrity Modeler contains a built-in service for configuration when introducing CI practices is to enable the integration of a management. It includes a weak optimistic locking mechanism version control tool in the modeling tool, but circumscribing model allowing multiple developers to collaborate on the same artefacts differencing and merging to the modeling tool itself. Indeed, this simultaneously. When multiple users are editing the same artefact, level of support is provided by multiple tools: Simulink, Rhapsody, the changes of one of them are visible to each of the others in Magic Draw, and Papyrus. real-time. Alternatively, there is an optimistic locking mechanism available, where these changes are not visible. Then, merges can 4.3.2 Building. BridgePoint provides integrated support for be performed automatically and their results manually edited to code generation. This functionality forms the “Translatable” part in resolve merge conflicts. “eXecutable Translatable UML” (xtUML). Models in xtUML can be LabVIEW includes a tool showing graphical differences between transformed to C, SystemC or C++ using included model compilers. model versions. VCSs, such as Git and SVN, can be used to keep These compilers are open source and can be customized. It is also track of different model versions. The differencing functionality of possible to create new compilers to translate models into differ- those tools can then be redirected to use the graphical difference ent programming languages. Changes to generated code are not available in LabVIEW. Merges can be performed automatically and propagated back to models. So, there is only support for one-way merge conflicts can be resolved manually. development and not for the round-trip from models to code and Magic Draw contains a built-in server for version control, it back. When the generated code is a complete application rather provides a model repository and supports collaboration through than a skeleton or a detached, individual subsystem, this is not branching and merging. Branching allows multiple developers necessarily an impediment to introducing CI practices. to work in parallel on the same project. A plug-in is available In Enterprise Architect, skeleton code can be generated from to support merging at model level. In case branching is not used, both Class diagrams and Interface models. More detailed code can concurrent changes directly on the mainline are prevented by means be generated from sequence-, activity-, and state machine diagrams. of pessimistic locking. The locks can be acquired at sub-model level, Several languages are supported, including C, C++, C#, and Java. i.e., parts of a model can be locked for editing. The locks can then Reverse engineering is also (partially) supported since some UML be released or maintained on commit. diagrams can be generated from code. Enterprise Architect Papyrus can be extended using plug-ins that are part of the includes a development environment where generated code can Eclipse Modeling Framework (EMF). The Collaborative Modeling be edited. This environment also supports typical functionalities initiative provides such plug-ins, in terms of collaboration support of a code editor such as debugging and profiling. Code generation for modeling in Eclipse, by using EMFCompare for the detection and reverse engineering can be combined and an option exists and merging of changes, EGit for distributed version control and to keep models and code synchronized. When generated code Gerrit for reviews of models. This allows developers to create a is updated due to a change in the model, the body of methods branch for a project, make changes and merge them into the main- is untouched, only their headers are changed such that previous line while staying on the model level. The included version control work is not undone. Although Enterprise Architect provides system EGit is an implementation of Git, incorporated in Eclipse. traceability matrices from requirements to models for requirements EMFCompare shows differences of changes between model ver- engineering, impact analysis for changes in models is not supported. sions from several views (graphical, textual, tabular). It can au- Some work has been done on creating impact analysis techniques tomatically merge changes, or in case of conflicts in three-way for any Enterprise Architect model, showing the potential of merges, allows the developer to choose the version to be integrated. third party solutions to solve this problem [17]. Rhapsody includes the tool DiffMerge, which can show graph- Integrity Modeler supports code generation from class and ical differences and automatically merge models or projects con- state-machine diagrams in several programming languages, includ- taining models. In case of any merge conflicts, the developer is ing Ada, C++, and Java. The generated code might be complete but shown a graphical difference between the versions and can resolve usually manual editing is required [22]. The tool includes function- the conflict by choosing one of the versions. The tool also supports ality to keep models and code synchronized in real-time when man- integration of version control systems ClearCase and SVN. ually altering the code. Furthermore, it supports impact analysis Simulink provides version control support through an integrated by letting the user define relationships between different modeling SVN instance but can also be used together with Git. This allows artefacts. These relationships can then be visualized to identify a project to be branched and thus models to be edited in parallel. potential model elements that need to be synchronized. Simulink contains an integrated tool for three-way model merging. Using additional code generators, C and Ada code can be gener- The tool automatically merges models and on conflict offers a choice ated from LabVIEW models. Templates on which the generations between the remote, base and local change. are based can be customized. Alternatively, the generated code can be customized after generation. Code generation is a one-way Summary. There are three main approaches for model versioning process, where the generated code is expected to be complete with in the evaluated tools. The first, locking, does not scale to large no manual editing required. The generators are designed to produce collaborative projects. The second, leaving versioning completely to code that can be integrated in a larger project. Magic Draw can generate code in several languages (Java, C++, scenario. We note that this type of synchronization functionality C#). In most cases, code generated from models will be skeletons focuses on models and code created in a single modeling tool. In and thus will be edited by developers to implement complete func- projects where multiple modeling tools are used, more challenges tionalities. There is also support for reverse engineering; models related to the synchronization of the different models can be ex- can be derived from code. Forward and reverse code engineering pected. This is mainly due to the limitation of impact analysis to is managed using Code engineering sets. These sets contain model assess only the impact of changes in models to models created in elements for which code is generated and conversely files from the same tool. which code is reversed to models. In addition, relationships be- tween model components can be defined. These can be visualized 4.3.3 Testing. To test models, BridgePoint provides a verifier. in different ways to show the impact of changes on the remaining This functionality forms the “eXecutable” part of xtUML. The ver- model artefacts in the project. ifier can be used to test models without the need to regenerate Papyrus supports code generation from UML models through source code. It executes the model itself and supports placement plug-ins. There are plug-ins available for the generation of C++ of breakpoints and inspection of variable values during the sim- and Java code from UML models, but it is also possible to create ulations. This can be useful for manual testing, but less so in CI custom code generators for other languages. Reverse engineering settings, where automated testing is preferred. is supported as part of the Papyrus Software Designer tooling, In Enterprise Architect, models can be simulated and test using it, class diagrams can be generated from Java classes and scripts can be defined to automatically test model elements. In these packages. Using the Papyrus Software Designer plug-in, models scripts, unit tests for Java (JUnit) or .NET (NUnit) can be called. A and generated code can be synchronized. Changes to the code are skeleton for these unit tests can automatically be generated from then propagated back to the model and changes in the model are class diagrams. By default, the tool supports the validation of incrementally applied to the code. models with respect to the UML syntax, but custom rules can be Rhapsody can generate code in C, C++, Java and, using a spe- added. Furthermore, validation and test scripts can be used to cific Rhapsody Developer version, also for Ada. This is done automate testing of models, whereas the simulation functionality incrementally, i.e., only new code is generated for modified model is mostly meant for debugging. elements. The generated code can be modified and changes are LabVIEW includes a framework for unit testing. Test cases can propagated back to the model. It is also possible to specify code be defined in the tool itself by defining input values and expected that should not be included in this round-trip, which could be use- output values for a specific unit under test. The tests can be executed ful for implementation-specific code that is not to be reused in in isolation or in a test suite. The tool includes a functionality other versions of a product. To see the impact of a change on the to track tests and the code it covers, automatically providing the other artefacts in the models, Rhapsody supports automated im- developer with code coverage information. In addition to this, pact analysis. The user configures the analysis by defining, among models can be validated using static code analysis rules, which can other things, the types of links to follow and their depth. Given the be customized for particular purposes. result of an impact analysis, it is up to the developer to manually Integrity Modeler contains a framework for automated test- co-evolve the impacted artefacts. ing. In it, test cases can be defined, triggered and their results The generation of C and C++ code from Simulink models is viewed. The test cases can also be grouped in sessions, allowing supported by additional tools that can be integrated in Simulink, their execution to be automated. such as Embedded Coder and Simulink Coder. Generated code is a In Magic Draw, models can be validated with respect to pre- complete program, not just skeleton code. Modifying the generated defined constraints or custom created constraints expressed in the code can be done at the level of the code generators, which can be Object Constraint Language (OCL). If the validation logic cannot be configured to replace code by custom snippets. This also means expressed in OCL, boolean constraints can be defined in Java. Ad- that the process of code generation is one-way; there is no support ditionally, unit tests can be defined to verify models or integrations. for propagating manual changes in the generated code back to JUnit is used to express test cases that can be executed using the the model. Simulink also contains a facility for automated impact build-in test framework. The framework also provides functionality analysis that can predict impacted elements in anticipation of a for checking the created program for memory leaks. particular change. Papyrus models can be validated with respect to predefined soundness constraints. Custom constraints can be defined in OCL. Summary. The basic functionality of generating skeleton code Validations can be performed on an entire model as well as on from models is present in each of the considered tools. The detail of parts of a model. Warnings or errors are shown in the models the created models dictates whether the entire application or only themselves after a validation. This validation is a static check, but skeleton code can be generated. This distinction usually influences UML models can also be executed, using the execution engine of the functionality regarding synchronization of models and code too. the Moka module. This can be used to manually test models, but This aspect is usually better supported in tools that just produce there is also support for automatic testing. The Papyrus Testing skeleton code than in tools that produce complete code and where Framework supports the automatic generation of unit tests from thus the generated code does not require manual editing. We have UML diagrams. The unit tests can be automatically tested using the seen that some tools contain functionality to assess the impact JUnit framework. of changes at model level, but that the implementations still rely Before code is generated in Rhapsody, a model-checker can be mostly on manual actions, which is not ideal in an automated build run to validate the model with respect to predefined and custom defined rules. Such custom rules have to be written in Java and applicable in fewer cases than the generally applicable batch mode can be used to check both the structural and the behavioral aspects construct. of the model. Included in the tool is also a functionality to simu- For Papyrus, code generation and model testing functionalities late models (or animate as is the used terminology for this tool). are packaged in Eclipse plug-ins. These are executable from the In addition, the tool can be integrated with other IBM Rational command line, which can be leveraged to include Papyrus in a CI tools for testing (Test RealTime) and quality assessment (Quality pipeline, for example by calling these plug-ins from scripts man- Manager). Furthermore, the tool includes a framework for the aged by a CI server such as Jenkins. Creating a CI pipeline for automatic generation of test cases. conventional Eclipse projects is a common practice, so it is not Similar to code generation, there exist additional tools for the expected that these particular tools would yield new problems. validation and testing of Simulink models. Simulink Test is a tool Rhapsody offers command line interfaces for code generation that supports creation and execution of test cases for models. Test and the DiffMerge tool. Using these commands, code can be cases can be defined to verify the models with respect to functional generated for specific components or for a project containing a constraints. It also provides an overview of failed and succeeded number of components. This allows for these tasks to be integrated test cases, similar to the dashboard of CI tools. in a CI pipeline where only models are checked in to the version control system and the application is generated. Summary. Most tools contain a unit testing mechanism. Some It is possible to create a CI pipeline using Jenkins to automat- implement their own and some use existing frameworks such as ically execute builds and tests in Simulink. Furthermore, the CI JUnit. Most tools also include model validation functionality, a tool can be configured to report on the success or failure of the check of the well-formedness of the models with respect to the automated tests. Such a process can be created using MATLAB, Git metamodel. Additionally, some tools are capable of simulating and Jenkins [8]. models, which is primarily useful for manual debugging. Next, we look at ways to automate builds and tests in the considered tools. Summary. With some effort, each modeling tool can be included 4.3.4 Automation. The BridgePoint editor is based on Eclipse in a CI pipeline. We have seen some examples of ready-made and its model compilers are implemented as Eclipse plug-ins. It is pipelines including some of the discussed tools. As long as the possible to run these from the command line and thus incorporate different approaches to automation can still be executed from a them as a build step in a CI pipeline. Similarly, the testing function- pipeline, there should be no impediments regarding combining ality incorporated in the verifier can be included in an automated automation for multiple modeling tools in one pipeline. process. Enterprise Architect offers the possibility to create analyzer 4.3.5 Evaluation Summary. Table 3 summarizes the evaluations scripts, these can be used to automate builds, tests, and other func- by scoring the different aspects in each tool as mature, standard, tionalities. The scripts can be created in the tool itself and allow for or immature. We define the thresholds for these scores, based on execution of the builds and tests from the command line. The scripts the identified aspects in Section 3, as follows. For integration, an can also be used to specify an output file to contain a generated immature level of support is considered an approach that does not report on the test results. Furthermore, they can be used to execute allow versioning of models, but e.g. utilizes pessimistic locking. the models and deploy the project, but that is out of the scope of A standard support would be one where models can be merged. our definition of CI. A mature level of support is considered when the tool also sup- Automation for CI can easily be achieved in Integrity Modeler ports the visualization and resolution of conflicts. For building, an using a Jenkins plug-in.3 The plug-in can detect changes in the built- immature level of support would be one where no code genera- in repository and can be configured to execute builds after such tion is possible. Code generation is considered standard support, a detection. Furthermore, it can retrieve the results of automated while mature support includes back-propagation of code changes tests, executed after the build. The availability of the Jenkins plug-in to models. For testing, immature level of support only includes signals a higher level of maturity with respect to CI processes than syntactical validation of models. Standard support involves also seen in other tools. verification of models using model testing. Support for testing is For LabVIEW, command line interfaces are available as open- considered mature when it includes unit tests for code and models. source.4 These allow the builds and tests to be executed by a CI For automation, tools are considered to provide immature support server, e.g. Jenkins. The test reports created by the tool can be if they do not feature explicit hooks to incorporate them in a CI stored as HTML files and as such be shown in Jenkins [9]. pipeline. A standard level of support includes the possibility to run Magic Draw supports extensibility by custom add-ins through the tool in batch mode and perform some actions. We require the its Open API. This API also allows the tool to be run in batch mode. presence of a command-line API to grant mature level of support This allows command line access to code generation and unit test for automation. execution, which makes it suitable to be used in a CI pipeline. Alter- natively, Magic Draw can also be integrated in other applications using its OSGi interfaces. Since this construct is Java-based, it is 5 DISCUSSION We have summarized the answers to our research questions by 3 https://wiki.jenkins.io/display/JENKINS/PTC+Integrity+Plugin Last access: June 4, listing relevant features for a modeling tool to be able to support 2018 CI practices in Table 1 and by summarizing the extent to which 4 https://github.com/JamesMc86/LabVIEW-CLI, retrieved: June 1, 2018 these features are present in current modeling tools in Table 3. Table 3: Aspects as supported by tools, scored by -, ◦, or +, de- Similarly, the selection of the tools contains a bias towards UML picting immature, standard, or mature support respectively. and SysML. Other languages or modeling paradigms may yield different impediments. Tools Other threats are related to our chosen research methodology. The evaluations are based on publicly available documentation Magic Draw BridgePoint Enterprise and research papers. This means that we did not experiment with Rhapsody Architect Simulink Integrity LabView Modeler Papyrus the tools themselves to assess the aspects. The advantage of this approach is that we avoid defining a scenario to evaluate the tools, which may not fit all tools or may accidentally favor some of them. Integration - - ◦ + + + + + On the other hand, the inherent threat of this approach is that Aspects Building ◦ + + ◦ + + + ◦ it does not highlight possible issues only visible when using the Testing ◦ + ◦ + ◦ + + ◦ evaluated tools in practice. Automation - ◦ + + ◦ ◦ + + 7 CONCLUSIONS AND FUTURE WORK In this paper, we identified relevant aspects of modeling tools to support CI practices. We then evaluated eight modeling tools and Regarding integration, most of the considered tools provide sup- assessed their levels of support for each of the aspects. In the port for differencing and merging at model level. They provide evaluated tools, we have seen different maturity levels of support representations in various formats of model differences and allow for the considered aspects. Overall, we found some challenges, developers to resolve merge conflicts at model level. The storage but no insurmountable impediments to introducing CI practices in of models and change history is usually managed by a VCS such MBD. as SVN or Git. For building, all tools provide some form of code The next step of our research consists in a set of interviews generation. Nonetheless, there is a great variability in the maturity with MBD practitioners working at different MBD maturity levels of what the environments can do after the code is generated. Some and in different companies. Practitioners will be asked to share tools automatically keep code and models synchronized whereas their views on introducing CI practices in MBD and their views on in others code generation is a one-way operation. The most chal- the impediments in their context. These interviews may uncover lenging aspect seems to be the synchronization of different models, hidden technical aspects that did not arise in this work, or bring up for which most of the tools include some level of change impact other, non-technical impediments, such as those briefly mentioned analysis support. Another challenge is the synchronization of mod- in Section 5. els when multiple modeling tools are used in the same software project and combined in the same pipeline. Testing is supported by each of the tools, but with different maturity levels. Finally, it 8 ACKNOWLEDGEMENTS is possible to automate at least parts of the build and testing pro- The authors would like to thank the industrial partners for their cesses for each of the tools. For some of the modeling tools such input in discussions about this work. This work is part of a project automated approaches are available, whereas for others it would supported by Software Center.5 require custom configuration. From the evaluations of the modeling tools, we did not find any REFERENCES theoretical obstacle in introducing CI practices in MBD. Some tools [1] Hessa Alfraihi and Kevin Lano. 2017. The Integration of Agile Development and Model Driven Development - A Systematic Literature Review. In Proceedings already have fairly good support for CI, and by cherry picking of the 5th International Conference on Model-Driven Engineering and Software features from other tools, they could be even more suitable for Development (MODELSWARD 2017). SCITEPRESS, 451–458. CI practices. In fact, in Section 4.3.4 we have mentioned some [2] Hessa Alfraihi and Kevin Lano. 2017. A Process for Integrating Agile Software Development and Model-Driven Development. In Proceedings of MODELS 2017 applications of CI in each of the tools Integrity Modeler, Lab- Satellite Event: FlexMDE. 412–417. VIEW, and Simulink. On the other hand, we foresee challenges [3] Kerstin Altmanninger, Martina Seidl, and Manuel Wimmer. 2009. A Survey on Model Versioning Approaches. International Journal of Web Information Systems in model synchronization and automation in projects involving (IJWIS) 5, 3 (2009), 271–304. multiple modeling tools. [4] Scott W Ambler. 2003. Agile Model Driven Development is Good Enough. IEEE Software 20, 5 (2003), 71–73. [5] Paul Baker, Shiou Loh, and Frank Weil. 2005. Model-Driven Engineering in a 6 THREATS TO VALIDITY Large Industrial Context—Motorola Case Study. In LNCS 3713. Springer, 476–491. In drafting both the list of relevant aspects and the list of considered [6] Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward Cun- ningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron tools there is an amount of subjectivity and possible bias. Tools Jeffries, and others. 2001. Manifesto for Agile Software Development. (2001). and aspects may be omitted or be listed but less relevant. Incom- http://agilemanifesto.org [7] Francis Bordeleau, Grischa Liebel, Alexander Raschke, Gerald Stieglbauer, and pleteness of the list of aspects was a threat to the validity of this Matthias Tichy. 2017. Challenges and Research Directions for Successfully Ap- work, since we aim to find impeding aspects. If crucial aspects plying MBE Tools in Practice. In Proceedings of the 20th International Conference were not included, we could not find the corresponding problems on Model Driven Engineering Languages and Systems (MODELS). 338–343. [8] Andy Campbell. 2015. The Other Kind of Continuous Integra- in the evaluated tools. To limit this risk, we gathered core rele- tion. (2015). https://blogs.mathworks.com/developer/2015/01/20/ vant aspects by investigating existing literature on the topics of the-other-kind-of-continuous-integration Retrieved: 2018-05-14. CI and MBD; furthermore, the lists were informally validated by two practitioners from industry with experience in MBD and CI. 5 www.software-center.se [9] Fredrik Edling. 2013. Using LabVIEW in a Continuous Integration Environment. [23] Erkuden Rios, Teodora Bozheva, Aitor Bediaga, and Nathalie Guilloreau. 2006. (2013). ftp://ftp.ni.com/pub/branches/northern region/nidays2013/presentations/ MDD Maturity Model: A Roadmap for Introducing Model-Driven Develop- sw cag using lv in cont integr environment.pdf Retrieved: 2018-06-01. ment. In Proceedings of the European Conference on Model Driven Architecture- [10] Jean-Marie Favre. 2004. Towards a Basic Theory to Model Model Driven En- Foundations and Applications. Springer, 78–89. gineering. In 3rd Workshop in Software Model Engineering, WiSME. Citeseer, [24] Douglas C Schmidt. 2006. Model-Driven Engineering. IEEE Computer 39, 2 262–271. (2006), 25. [11] Martin Fowler. 2006. Continuous Integration. (2006). https://martinfowler.com/ [25] Bran Selic. 2003. The Pragmatics of Model-Driven Development. IEEE software articles/continuousIntegration.html 20, 5 (2003), 19–25. [12] Jokin Garcia. 2018. Continuous Model-Driven Engineering. (2018). https: [26] Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous //modeling-languages.com/continuous-model-driven-engineering/ Retrieved: Integration, Delivery and Deployment: a Systematic Review on Approaches, 2018-05-14. Tools, Challenges and Practices. IEEE Access 5 (2017), 3909–3943. [13] Jokin Garcia and Jordi Cabot. 2018. Stepwise Adoption of Continuous Delivery [27] Daniel Ståhl and Jan Bosch. 2013. Experienced Benefits of Continuous Integration in Model-Driven Engineering – Extended Abstract. DEVOPS (2018). in Industry Software Product Development: A Case Study. In The 12th IASTED [14] Vicente Garcı́a-Dı́az, Jordán Pascual Espada, Edward Rolando Núnez-Valdéz, G International Conference on Software Engineering (Innsbruck, Austria, 2013). 736– Pelayo, B Cristina Bustelo, and Juan Manuel Cueva Lovelle. 2016. Combining the 743. Continuous Integration Practice and the Model-Driven Engineering Approach. [28] Daniel Ståhl, Torvald Mårtensson, and Jan Bosch. 2017. Continuous Practices Computing and Informatics 35, 2 (2016), 299–337. and DevOps: Beyond the Buzz, What Does It All Mean?. In Software Engineering [15] Sebastian Hansson, Yu Zhao, and Håkan Burden. How MAD are we? Empirical and Advanced Applications (SEAA), 2017 43rd Euromicro Conference on. IEEE, Evidence for Model-driven Agile Development. In Proceedings of XM 2014, 3rd 440–448. Extreme Modeling Workshop, Vol. 1239. 2–11. [29] Stavros Stavru, Iva Krasteva, and Sylvia Ilieva. 2013. Challenges of Model-driven [16] John Hutchinson, Jon Whittle, Mark Rouncefield, and Steinar Kristoffersen. 2011. Modernization-An Agile Perspective.. In MODELSWARD. 219–230. Empirical Assessment of MDE in Industry. In Proceedings of the 33rd International [30] Henrik Steudel, Regina Hebig, and Holger Giese. 2012. A Build Server for Conference on Software Engineering (ICSE). IEEE, 471–480. Model-Driven Engineering. In Proceedings of the 6th International Workshop on [17] Melanie Langermeier, Christian Saad, and Bernhard Bauer. 2014. Adaptive Multi-Paradigm Modeling. ACM, 67–72. Approach for Impact Analysis in Enterprise Architectures. In International Sym- [31] Nenad Ukić, Pál L Pályi, Marijan Zemljić, Domonkos Asztalos, and Ivan Markota. posium on Business Modeling and Software Design. Springer, 22–42. 2011. Evaluation of Bridgepoint Model-Driven Development Tool in Distributed [18] Grischa Liebel, Nadja Marko, Matthias Tichy, Andrea Leitner, and Jörgen Hans- Environment. In Workshop on Information and Communication Technologies son. 2016. Model-Based Engineering in the Embedded Systems Domain: an conjoint with 19th International Conference on Software, Telecommunications and Industrial Survey on the State-of-Practice. Software & Systems Modeling 17, 1 Computer Networks, SoftCOM 2011. (2016), 91–113. [32] Ragnhild Van Der Straeten, Tom Mens, and Stefan Van Baelen. 2008. Challenges [19] Torvald Mårtensson, Daniel Ståhl, and Jan Bosch. 2017. Continuous Integration in Model-Driven Software Engineering. In International Conference on Model Impediments in Large-Scale Industry Projects. In Proceedings of the 2017 IEEE Driven Engineering Languages and Systems. Springer, 35–47. International Conference on Software Architecture (ICSA). IEEE, 169–178. [33] Jon Whittle, John Hutchinson, Mark Rouncefield, Håkan Burden, and Rogardt [20] Ade Miller. 2008. A Hundred Days of Continuous Integration. In Agile. IEEE, Heldal. 2013. Industrial Adoption of Model-Driven Engineering: Are the Tools 289–293. Really the Problem?. In International Conference on Model Driven Engineering [21] Parastoo Mohagheghi, Wasif Gilani, Alin Stefanescu, and Miguel A. Fernandez. Languages and Systems. Springer, 1–17. 2013. An Empirical Study of the State of the Practice and Acceptance of Model- [34] Yuefeng Zhang and Shailesh Patel. 2011. Agile Model-Driven Development in Driven Engineering in Four Industrial Cases. Empirical Software Engineering 18, Practice. IEEE software 28, 2 (2011), 84–91. 1 (01 Feb 2013), 89–116. [22] David Norfolk. 2015. PTC Integrity Modeler — a Standards-Based Tool for Systems and Software Engineering. Technical Report.