Introducing MDML - A Domain-specific Modelling Language for Automotive Measurement Devices Christian Burghard, Gerald Stieglbauer, Robert Korošec1 Abstract. The method of model-based mutation testing has was used to generate test procedures, based on a model of the recently been introduced in the AVL List GmbH to automat- MD’s functional behaviour. UML state diagrams were chosen ically test the various measurement devices within the com- as the modelling language for the device behaviour. However, pany’s portfolio. However, the initial approach of using UML this initial approach turned out to be non-satisfying in prac- as a modelling language turned out to be non-satisfying. In tice. An internal study revealed that UML was rejected for this paper, we introduce a textual domain-specific language various technical and usability-related reasons. In this paper, for the sole purpose of modelling the behaviour of measure- we introduce a text-based domain-specific language (DSL) as ment devices. Furthermore, we examine the technical differ- an alternative to UML state diagrams. We examine technical ences to UML as well as the usability-related aspects, which aspects like semantics and expressiveness, as well as usabil- determine the acceptance of the language by its end users. ity aspects like simplicity of model creation and intuitive un- derstandability. We finally compare this DSL to UML in the 1 Introduction context of our industrial case study and argue why the DSL seems to fit better and increases the acceptance rate by the end users. The rest of this paper is structured as follows: Chapter 1.1 examines the MD testing process as it was before the introduction of model-based testing. Chapter 1.2 describes the former UML-based approach to model-based testing, as well as the problems of this initial approach. In Chapter 2 we describe the design approach and implementation of our DSL. Finally, the initial reactions of our test users, as well as a comparison to UML are summarized in Chapter 3. 1.1 Previous testing approach Figure 1. Portfolio AVL Instrumentation and Test Systems AVL vehicle and powertrain testbeds are typically comprised of many different devices, such as measurement devices. The AVL Instrumentation and Test Systems provides a prod- individual MDs are controlled by a test automation software uct portfolio, which is applied in the automotive industry for called PUMA Open [4]. PUMA Open runs on a standard PC, integration and testing activities within the field of power- which is connected to the devices. The MDs are tested by a train engineering (see Figure 1). Instruments (measurement group of test engineers. A server-sided abstraction layer al- devices, MDs) are an essential part of test systems, required lows them to handle each MD in the form of a software ob- for measuring specific quantities of the unit under test (air and ject which abstracts the device’s functionality [5]. To test the fuel consumption, emissions, combustion, etc.). The measure- integration of MDs into the automation system, simulation ment devices are integrated into a test automation system models virtualize the functionality of the MDs. This virtual- that orchestrates the different measurement devices partici- ization is required to avoid cost-intensive hardware and enable pating in a test activity. An important part of the MD devel- device variant testing. For this purpose the test engineers use opment process is testing of the control interfaces and of the a Testbed Simulator (TBSimu) which provides the same soft- devices’ functional operation states. To make the development ware interface as the actual devices. For the automated testing of testing procedures more effective and efficient, AVL initi- process, AVL uses an internal Test Automation Framework ated the introduction of model-based mutation testing into (TAF), which allows the execution of various NUnit Tests the process. The MoMuT test case generation tool [1, 2, 3] [6] on each MD’s software interface. Traditionally, the NUnit 1 AVL List GmbH, Hans-List-Platz 1, 8020 Graz, tests are manually written by the test engineers. In most cases email: {Christian.Burghard, Gerald.Stieglbauer, they use the MD’s manual as a reference to define a set of test Robert.Korosec}@avl.com cases. Currently, there is no way to objectively measure the test coverage, which has to be estimated empirically by the test engineer. In the event of a specification change, e.g. by creating a new MD variant, the test suite for a MD has to be manually revised. This process is error-prone and very time- consuming. 1.2 Introduction of a model-based testing approach To make device testing more effective and efficient, a model- based testing approach was investigated in the TRUFAL project2 [3]. The functional operation states of the MDs can generally be described in the form of state machines. As an initial trial, a UML State Diagram of the AVL489 particle emission measurement device was developed [5]. The defini- tion of the necessary subset of UML model elements and their semantics [1] was reused from the MOGENTES project3 . The UML-based device model of AVL489 was engineered by a test user with the help of various domain experts and Figure 2. TRUCONF testing toolchain for measurement served as an input for the MoMuT test case generator via devices. The impact of our DSL will be mostly confined to the an intermediate language called OOAS [7]. MoMuT gener- area indicated by the red dashed line. ated a series of abstract testing procedures in the so-called Aldebaran file format [8]. These abstract test procedures were • The DSL has to provide an efficient way to create MD ultimately converted to concrete NUnit tests which are exe- models in a compact and expressive form with semantically cutable on the TAF. However, it turned out that the typical well defined model elements. test engineer has a very weak background in UML seman- • The DSL has to enable the generation of meaningful test tics. In addition, currently available tools hardly provide any cases based on partial MD models. More specifically, the support in guiding the users in using the right UML seman- test case generation needs to yield meaningful results, even tics [1] for their intended purpose. On the other hand, even if the model only contains a subset of all inputs allowed a UML-experienced modeller had a hard time obtaining the by the system under test (SUT). But for all allowed inputs necessary information because UML’s abstraction approach the SUT’s output has to conform to the output specified in did not mirror the test engineers’ intuitive way of thinking, the model. Compare the Input-Output Conformance (ioco) which differs from standard UML state diagrams and turned relation [9, 2]. (Currently, our test cases only test the pres- out not to be sufficiently represented by UML elements. ence of required outputs and not the absence of forbidden ones, but the testing mechanism can be modified to test for 2 Introducing a domain-specific language full ioco.) Furthermore, distributing an MD’s functionality over several models should greatly facilitate variant testing. As we faced these challenges, we decided to create a domain- specific language which is specifically tailored to MD be- haviour modelling. This DSL is being designed in the ongoing 2.1 Language design TRUCONF project4 . TRUCONF is a follow-up to TRUFAL In accordance with the test engineers an initial approach of and will extend the automated test case generation to incor- developing a textual DSL based on Gherkin [10] was evalu- porate the functional and non-functional aspects of MDs. An ated. This language is designed for behaviour-driven devel- overview of the TRUCONF toolchain is given in Figure 2. opment and allows simply structured scenario descriptions The non-functional cost models (e.g. CPU load, memory con- in natural language. Colombo et al. have previously used sumption, data rates, etc.) will be mostly obtained via model Gherkin to generate test cases based on state machine models learning. (This is currently subject of research and would thus [11]. In their approach the authors describe each transition of go beyond the scope of this paper.) Therefore, the DSL cur- their state machine with a Gherkin scenario. For each sce- rently focusses on the modelling of functional behaviour. Such nario they use the keyword Given to denote the current state, a language has to satisfy the following requirements: the keyword When for external triggers and the keyword Then for the next state. In our case the external triggers represent • The DSL has to be intuitively understandable to its end the AK commands [12] that are used to control the device users, even if they have no prior modelling experience. This remotely. For example: seems to be the most important requirement for the DSL’s industrializability since the TRUFAL project has shown Scenario : Go from Pause t o Standby that ignoring this leads to a strong rejection of the sug- Given t h e d e v i c e i s i n S t a t e Pause gested model-based testing approach by the test engineers. When I s e n d t h e s i g n a l STBY Then t h e d e v i c e s h o u l d be i n S t a t e Standby . 2 https://trufal.wordpress.com/ (2016.07.29) 3 http://www.mogentes.eu/ (2016.07.29) As a basis for the language design, a Gherkin model of 4 http://truconf.ist.tugraz.at/ (2016.07.29) AVL489 was developed. After that the Gherkin-based lan- guage design was iteratively refined and optimized in close This is an important improvement, since the mere introduc- interaction with its end users to describe the MD in a concise tion to the UML IDE Visual Paradigm [15] and the used UML and easily understandable way. The explicit naming of sce- semantics took significantly longer. One test user was tasked narios was omitted because it was deemed redundant. The with the modelling of several MDs over an extended period natural-language formulation of current state, trigger and of time. Within a week, he created extensive models of all the next state was replaced by a compact formal notation. For in- devices that were assigned to him. The models were based stance, since there could be many transitions originating from on the behaviour specification in the MD manuals and cross- one current state, the scenarios could be grouped in blocks referenced with the behaviour of the virtual TBSimu devices. with a common Given statement. The semantics of the Then However, a final judgement of the model quality can only be statement was slightly strengthened to describe the MDs re- made when the generated test suites are applied to the real action, rather than a postcondition. With these changes the devices. model length was reduced by more than half in the very first On the other hand, the test users expressed the wish for design iteration. After some additional syntactic fine-tuning several concrete IDE features: It would be helpful to provide we arrived at the following notation: the test engineers with some kind of graphical feedback to assess the degree of completion of the current model at first given D e v i c e S t a t e = Pause { glance. Here several aspects of UML state diagrams could be when A c t i o n = STBY then D e v i c e S t a t e −> Standby ; when A c t i o n = SMES then D e v i c e S t a t e −> Measure ; reused, but in a use case-tailored manner and rather as an when . . . } optional view with limited editing features, than a first class input language. Furthermore, the semantical correctness of This example includes the same transition as the previous one, an MDML model is not fully defined by its grammar. Thus as well as another transition to the state Measure and also there is a strong need for a model validation feature. For ex- hints at other possible transitions leaving the state Pause. If ample it has to be checked if the decision tree allows outgo- the state machine encompasses several orthogonal state vari- ing transitions for all possible states and that no dead ends ables, the given blocks can be cascaded to form a decision are produced. The validation result can be given back to the tree. To reflect the high degree of purpose orientation, we user in the form of a graphical or tabular view. Moreover named our language ”Measurement Device Modelling Lan- the validator has to ensure that the model does not contain guage” (MDML). In this paper, we merely show a glimpse of multiple when ... then statements for the same input that MDML, as a detailed language description would go beyond are allowed under the same conditions, which would produce the scope of this work. non-deterministic behaviour. 2.2 Implementation 3.2 Selected key differences to UML To complement our purpose-driven language design, we MDML aims to improve on the practical and technical short- wanted to supply the test engineers with an editor that sup- comings of the previously used UML approach. However, ports them in their concrete work as extensively as possi- MDML’s language design is still in progress and it does not ble. Thus we decided to implement a plugin for the widely yet support timed behaviours. A selection of key differences used Eclipse IDE [13] by means of the Xtext framework [14]. between UML and MDML in the context of our use case is Xtext automatically creates an Eclipse-plugin from a gram- given below: mar definition. This plugin provides a parser, as well as syntax highlighting and simple syntactic quick fixes out-of-the-box. Moreover, it highly facilitates the implementation of other features like auto-completion, code formatting, and tool tips. Such features greatly assist the test engineer in building a syntactically and semantically correct model and improve on the perceived lack of user-friendliness during the TRUFAL Figure 3. The Manual/Remote sub-state machine of AVL489. project. As we did with our language design, we put the end users in 3.2.1 Concrete examples the very center of our tool integration process. We encourage the test engineers to gain user experience and give feedback 1. If the state machine contains several parallel sub-state ma- that will allow us to iteratively improve the IDE. chines, the UML semantics requires the user to incorporate hidden boolean variables for communication between these sub-state machines. This issue is best described in the form 3 Results of an example: Figure 3 shows the Manual/Remote sub-state machine of the AVL489 device. The MD can only receive 3.1 User feedback commands from PUMA when it is in the state Remote. After its implementation, a prototype MDML IDE was made Thus all the inputs that change the states of a parallel available to the test engineers, as well as a trial group who sub-state machine (e.g. Standby, Measure, etc.) can only had no previous involvement in MD development and testing. be accepted if this condition is fulfilled. However, the UML All users agreed that MDML is very comprehensible and easy semantics does not allow for direct communication between to learn. All test users were able to generate an initial mea- the respective sub-state machines. Instead, the transition surement device model based on its manual within an hour. guards depend on boolean variables that communicate the respective states. These variables have to be explicitly set to come up with a clearly defined representation that mirrors in the states’ entry actions. For example, in the entry ac- the intuitive timing semantics of UML state diagrams. tion of the state Manual, the Manual variable is set to true, while it is set to false in Remote. This appears somewhat 3.3 Conclusion and outlook strange as it reads like ”When you are in Manual, then be- come Manual”. These internal variables are not visible at We have created a language design that proved to be highly first glance, so they reduce the readability and somewhat understandable to test engineers and is very easy to learn. undermine the graphical nature of the model. In MDML Their current feedback is quite promising and their involve- this shortcoming does not occur, as the decision tree can ment in the DSL design process has helped to increase the be made directly dependent on all state variables present DSL’s acceptance. While the textual model representation in the model, for example: gives the user great control over small details, there is still a need for a graphical representation to show the big pic- given C o n n e c t i o n S t a t e = Remote { ture. The implementation of both a textual and a graphi- given D e v i c e S t a t e = Standby { when . . . } cal representation are important goals of our ongoing TRU- given . . . } CONF project. Furthermore, the Eclipse-based IDE will be extended to encompass a validation of the MDML model in 2. In addition to the state diagram, the UML approach re- terms of coverage and unambiguousness. While MDML solved quires several class diagrams to describe the interface be- several issues of the UML approach, its language design is tween the system under test and the environment, which still incomplete. The most important addition will be a syn- explicitly defines all possible inputs and outputs. In MDML tax and semantics for timing behaviour. This and other im- this information is replaced by a short header segment that provements are subject of our current work. Most importantly, defines the state variables and inputs in the following way: though, MDML will ultimately have to prove its worth for in- dustrial test case generation. An upcoming publication will public statevar D e v i c e S t a t e { Pause , Standby , describe the conversion of MDML into the intermediate lan- . . . } = Pause ; input A c t i o n {SPAU, STBY, . . . } ; guage OOAS and the test case generation process. State variables are defined by their name, their range and REFERENCES their initial state. The keyword public denotes that its current state is visible to the environment. Input symbols [1] W. Krenn, R. Schlick, and B. K. Aichernig, “Mapping UML to labeled transition systems for test-case generation: A trans- are explicitly defined and can be grouped into named input lation via object-oriented action systems,” vol. 6286 LNCS, channels, e.g. ”Action”. Currently, MDML does not allow pp. 186–207, 2010. the explicit definition of outputs, but this feature can be [2] E. Joebstl, Model-Based Mutation Testing with Constraint added, should the need arise. and SMT Solvers. Dissertation, Graz University of Technol- ogy, 2014. [3] B. K. Aichernig, J. Auer, E. Jöbstl, R. Korošec, W. Krenn, 3.2.2 Maintainability R. Schlick, and B. V. Schmidt, “Model-based mutation testing of an industrial measurement device,” vol. 8570 LNCS, pp. 1– The debugging and fine-tuning of the UML model of AVL489 19, 2014. [4] “AVL PUMA Open Automation Platform.” https: took up to 40 hours. This is due to the fact that the accurate //www.avl.com/-/avl-puma-open-automation-platform. representation of all small details turned out to be difficult (2016.07.29). and sometimes required a complete model revision. Due to [5] J. Auer, Automated Integration Testing of Measurement De- the improvements mentioned above and to the guidance that vices. Bachelors thesis, Graz University of Technology, 2013. our MDML IDE aims to provide, we hope to greatly reduce [6] “NUnit Framework.” http://www.nunit.org/. (2016.07.29). [7] S. Tiran, “The Argos Manual,” tech. rep., 2012. the necessary effort. It is also worth mentioning that all el- [8] “Aldebaran manual page.” http://cadp.inria.fr/man/ ements in an MDML model are immediately visible to the aldebaran.html. (2016.07.29). user while important aspects of a UML model can be hidden [9] J. Tretmans, “Test Generation with Inputs, Outputs and in various property menus of the respective editor. Hidden Repetitive Quiescence,” Software-Concepts and Tools, vol. 3, pp. 103–120, 1996. information can be a desirable feature under some circum- [10] “Gherkin.” https://cucumber.io/docs/reference#gherkin. stances, for example in a graphical representation designed to (2016.07.29). show an overview, but it can also pose a great challenge to an [11] C. Colombo, M. Micallef, and M. Scerri, “Verifying Web Ap- inexperienced user. plications: From Business Level Specifications to Automated Model-Based Testing,” Electronic Proceedings in Theoretical Computer Science, vol. 141, no. Mbt, pp. 14–28, 2014. 3.2.3 Timing behaviour [12] K. Jogun, “A Universal Interface for the Integration of Emis- sions Testing Equipment Into Engine Testing Automation UML represents timing behaviour in a very intuitive man- Systems: The VDA-AK SAMT-Interface,” SAE Technical ner. Although timing is an important part of most MDs’ be- Paper, 1994. [13] “Eclipse IDE.” http://www.eclipse.org/. (2016.07.29). haviour, MDML is currently lacking a representation of timed [14] “Xtext Framework.” https://eclipse.org/Xtext/. transitions. At the moment we treat timed actions as a kind (2016.07.29). of input (e.g. ”the user waits for 20 seconds and the device [15] “Visual Paradigm.” https://www.visual-paradigm.com. reacts to the user’s inaction”). However, this leaves a gap in (2016.07.29). MDML’s semantics, as it is not defined at which point the time frame starts. An important part of our current work is