The method of model-based mutation testing has recently been introduced in the AVL List GmbH to automatically test the various measurement devices within the company's portfolio. However, the initial approach of using UML as a modelling language turned out to be non-satisfying. In this paper, we introduce a textual domain-speci c language for the sole purpose of modelling the behaviour of measurement devices. Furthermore, we examine the technical di erences to UML as well as the usability-related aspects, which determine the acceptance of the language by its end users.
AVL Instrumentation and Test Systems provides a
product portfolio, which is applied in the automotive industry for
integration and testing activities within the eld of
powertrain engineering (see Figure 1). Instruments (measurement
devices, MDs) are an essential part of test systems, required
for measuring speci c quantities of the unit under test (air and
fuel consumption, emissions, combustion, etc.). The
measurement devices are integrated into a test automation system
that orchestrates the di erent measurement devices
participating in a test activity. An important part of the MD
development process is testing of the control interfaces and of the
devices' functional operation states. To make the development
of testing procedures more e ective and e cient, AVL
initiated the introduction of model-based mutation testing into
the process. The MoMuT test case generation tool [
The rest of this paper is structured as follows: Chapter 1.1 examines the MD testing process as it was before the introduction of model-based testing. Chapter 1.2 describes the former UML-based approach to model-based testing, as well as the problems of this initial approach. In Chapter 2 we describe the design approach and implementation of our DSL. Finally, the initial reactions of our test users, as well as a comparison to UML are summarized in Chapter 3. 1.1
AVL vehicle and powertrain testbeds are typically comprised
of many di erent devices, such as measurement devices. The
individual MDs are controlled by a test automation software
called PUMA Open [
To make device testing more e ective and e cient, a
modelbased testing approach was investigated in the TRUFAL
project2 [
The UML-based device model of AVL489 was engineered
by a test user with the help of various domain experts and
served as an input for the MoMuT test case generator via
an intermediate language called OOAS [
As we faced these challenges, we decided to create a domainspeci c language which is speci cally tailored to MD behaviour modelling. This DSL is being designed in the ongoing TRUCONF project4. TRUCONF is a follow-up to TRUFAL and will extend the automated test case generation to incorporate the functional and non-functional aspects of MDs. An overview of the TRUCONF toolchain is given in Figure 2. The non-functional cost models (e.g. CPU load, memory consumption, data rates, etc.) will be mostly obtained via model learning. (This is currently subject of research and would thus go beyond the scope of this paper.) Therefore, the DSL currently focusses on the modelling of functional behaviour. Such a language has to satisfy the following requirements: The DSL has to be intuitively understandable to its end users, even if they have no prior modelling experience. This seems to be the most important requirement for the DSL's industrializability since the TRUFAL project has shown that ignoring this leads to a strong rejection of the suggested model-based testing approach by the test engineers. 2 https://trufal.wordpress.com/ (2016.07.29) 3 http://www.mogentes.eu/ (2016.07.29) 4 http://truconf.ist.tugraz.at/ (2016.07.29)
The DSL has to provide an e cient way to create MD models in a compact and expressive form with semantically well de ned model elements.
The DSL has to enable the generation of meaningful test
cases based on partial MD models. More speci cally, the
test case generation needs to yield meaningful results, even
if the model only contains a subset of all inputs allowed
by the system under test (SUT). But for all allowed inputs
the SUT's output has to conform to the output speci ed in
the model. Compare the Input-Output Conformance (ioco)
relation [
In accordance with the test engineers an initial approach of
developing a textual DSL based on Gherkin [
Given the d e v i c e i s i n S t a t e Pause When I send the s i g n a l STBY
Then the d e v i c e s h o u l d be i n S t a t e Standby . As a basis for the language design, a Gherkin model of AVL489 was developed. After that the Gherkin-based language design was iteratively re ned and optimized in close interaction with its end users to describe the MD in a concise and easily understandable way. The explicit naming of scenarios was omitted because it was deemed redundant. The natural-language formulation of current state, trigger and next state was replaced by a compact formal notation. For instance, since there could be many transitions originating from one current state, the scenarios could be grouped in blocks with a common Given statement. The semantics of the Then statement was slightly strengthened to describe the MDs reaction, rather than a postcondition. With these changes the model length was reduced by more than half in the very rst design iteration. After some additional syntactic ne-tuning we arrived at the following notation: given D e v i c e S t a t e = Pause f when Action = STBY then D e v i c e S t a t e > Standby ; when Action = SMES then D e v i c e S t a t e > Measure ; when . . . g This example includes the same transition as the previous one, as well as another transition to the state Measure and also hints at other possible transitions leaving the state Pause. If the state machine encompasses several orthogonal state variables, the given blocks can be cascaded to form a decision tree. To re ect the high degree of purpose orientation, we named our language "Measurement Device Modelling Language" (MDML). In this paper, we merely show a glimpse of MDML, as a detailed language description would go beyond the scope of this work. 2.2
To complement our purpose-driven language design, we
wanted to supply the test engineers with an editor that
supports them in their concrete work as extensively as
possible. Thus we decided to implement a plugin for the widely
used Eclipse IDE [
As we did with our language design, we put the end users in the very center of our tool integration process. We encourage the test engineers to gain user experience and give feedback that will allow us to iteratively improve the IDE. 3 3.1
After its implementation, a prototype MDML IDE was made
available to the test engineers, as well as a trial group who
had no previous involvement in MD development and testing.
All users agreed that MDML is very comprehensible and easy
to learn. All test users were able to generate an initial
measurement device model based on its manual within an hour.
This is an important improvement, since the mere
introduction to the UML IDE Visual Paradigm [
On the other hand, the test users expressed the wish for several concrete IDE features: It would be helpful to provide the test engineers with some kind of graphical feedback to assess the degree of completion of the current model at rst glance. Here several aspects of UML state diagrams could be reused, but in a use case-tailored manner and rather as an optional view with limited editing features, than a rst class input language. Furthermore, the semantical correctness of an MDML model is not fully de ned by its grammar. Thus there is a strong need for a model validation feature. For example it has to be checked if the decision tree allows outgoing transitions for all possible states and that no dead ends are produced. The validation result can be given back to the user in the form of a graphical or tabular view. Moreover the validator has to ensure that the model does not contain multiple when ... then statements for the same input that are allowed under the same conditions, which would produce non-deterministic behaviour. 3.2
MDML aims to improve on the practical and technical shortcomings of the previously used UML approach. However, MDML's language design is still in progress and it does not yet support timed behaviours. A selection of key di erences between UML and MDML in the context of our use case is given below: 1. If the state machine contains several parallel sub-state machines, the UML semantics requires the user to incorporate hidden boolean variables for communication between these sub-state machines. This issue is best described in the form of an example: Figure 3 shows the Manual/Remote sub-state machine of the AVL489 device. The MD can only receive commands from PUMA when it is in the state Remote. Thus all the inputs that change the states of a parallel sub-state machine (e.g. Standby, Measure, etc.) can only be accepted if this condition is ful lled. However, the UML semantics does not allow for direct communication between the respective sub-state machines. Instead, the transition guards depend on boolean variables that communicate the respective states. These variables have to be explicitly set in the states' entry actions. For example, in the entry action of the state Manual, the Manual variable is set to true, while it is set to false in Remote. This appears somewhat strange as it reads like "When you are in Manual, then become Manual". These internal variables are not visible at rst glance, so they reduce the readability and somewhat undermine the graphical nature of the model. In MDML this shortcoming does not occur, as the decision tree can be made directly dependent on all state variables present in the model, for example: given C o n n e c t i o n S t a t e = Remote f given D e v i c e S t a t e = Standby f
when . . . g given . . . g 2. In addition to the state diagram, the UML approach requires several class diagrams to describe the interface between the system under test and the environment, which explicitly de nes all possible inputs and outputs. In MDML this information is replaced by a short header segment that de nes the state variables and inputs in the following way: public statevar D e v i c e S t a t e fPause , Standby , . . . g = Pause ; input Action fSPAU, STBY, . . . g ; State variables are de ned by their name, their range and their initial state. The keyword public denotes that its current state is visible to the environment. Input symbols are explicitly de ned and can be grouped into named input channels, e.g. "Action". Currently, MDML does not allow the explicit de nition of outputs, but this feature can be added, should the need arise. 3.2.2
The debugging and ne-tuning of the UML model of AVL489 took up to 40 hours. This is due to the fact that the accurate representation of all small details turned out to be di cult and sometimes required a complete model revision. Due to the improvements mentioned above and to the guidance that our MDML IDE aims to provide, we hope to greatly reduce the necessary e ort. It is also worth mentioning that all elements in an MDML model are immediately visible to the user while important aspects of a UML model can be hidden in various property menus of the respective editor. Hidden information can be a desirable feature under some circumstances, for example in a graphical representation designed to show an overview, but it can also pose a great challenge to an inexperienced user. 3.2.3
UML represents timing behaviour in a very intuitive manner. Although timing is an important part of most MDs' behaviour, MDML is currently lacking a representation of timed transitions. At the moment we treat timed actions as a kind of input (e.g. "the user waits for 20 seconds and the device reacts to the user's inaction"). However, this leaves a gap in MDML's semantics, as it is not de ned at which point the time frame starts. An important part of our current work is to come up with a clearly de ned representation that mirrors the intuitive timing semantics of UML state diagrams. We have created a language design that proved to be highly understandable to test engineers and is very easy to learn. Their current feedback is quite promising and their involvement in the DSL design process has helped to increase the DSL's acceptance. While the textual model representation gives the user great control over small details, there is still a need for a graphical representation to show the big picture. The implementation of both a textual and a graphical representation are important goals of our ongoing TRUCONF project. Furthermore, the Eclipse-based IDE will be extended to encompass a validation of the MDML model in terms of coverage and unambiguousness. While MDML solved several issues of the UML approach, its language design is still incomplete. The most important addition will be a syntax and semantics for timing behaviour. This and other improvements are subject of our current work. Most importantly, though, MDML will ultimately have to prove its worth for industrial test case generation. An upcoming publication will describe the conversion of MDML into the intermediate language OOAS and the test case generation process.