=Paper=
{{Paper
|id=Vol-3071/paper15
|storemode=property
|title=Towards Mutation Testing of Simulink Models
|pdfUrl=https://ceur-ws.org/Vol-3071/paper15.pdf
|volume=Vol-3071
|authors=Joanna Kisaakye,Onur Kilincceker,Serge Demeyer
|dblpUrl=https://dblp.org/rec/conf/benevol/KisaakyeKD21
}}
==Towards Mutation Testing of Simulink Models==
https://ceur-ws.org/Vol-3071/paper15.pdf
Towards Mutation Testing of Simulink Models
Joanna Kisaakye1 , Onur Kilincceker1,2 and Serge Demeyer1,2
1
Department of Computer Science, University of Antwerp, 2020 Antwerp, Belgium
2
Flanders Make vzw, Belgium
Abstract
Mutation testing offers stronger test adequacy than conventional structural coverage metrics. It employs
mutants, which can mimic the real software faults, to measure the fault detection capacity of a given test
suite. Mutation testing has been shown to work on an industrial scale for embedded systems, however
only for unit tests. Since it is common practice to use model-based design to analyze, and test embedded
software, it is worthwhile to investigate mutation testing in the context of MATLAB/Simulink. In this
paper, we propose a framework for mutation testing of Simulink models. We validate the framework by
means of a line follower robot constructed with Lego/Mindstorm and modelled using a series of Simulink
models. This study also addresses possible open research and development challenges.
Keywords
Mutation Testing, Simulink Models, Embedded Software, Model-based Design
1. Introduction
Software testing is an indispensable activity of modern software development to measure
software quality. This activity needs to be planned and executed carefully to avoid possible
software related failures. Code based coverage metrics are normally used to assess adequacy of
software testing methods, but only show whether a test touches the unit under test and not
whether the test reveals a defect. Today’s complex software systems demand not only practical
and effective methods but also corresponding coverage metrics. Mutation coverage is utilized
to address these concerns as being the state-of-the-art criterion for evaluating the strength of a
software test suite [1].
Mutation testing employs mutation operators to obtain mutants of the software, which are
faulty versions of the software and can mimic the real faults [2, 3]. Therefore, a detected fault
refers to a “killed” while an undetected fault refers to a “live” mutant for corresponding software.
To assess adequacy, the software testing methods evaluated with respect to mutation coverage
resulted from the application of mutation testing. Mutation coverage is simply the ratio of
killed mutants to total number of mutants neglecting the equivalent mutants that behave as the
original (fault-free) software and are very hard to kill.
BENEVOL’21: The 20th Belgium-Netherlands Software Evolution Workshop, December 07–08, 2021, ’s-Hertogenbosch
(virtual), NL
Envelope-Open Joanna.Kisaakye@student.uantwerpen.be (J. Kisaakye); onur.kilincceker@uantwerpen.be (O. Kilincceker);
serge.demeyer@uantwerpen.be (S. Demeyer)
GLOBE https://www.uantwerpen.be/en/staff/onur-kilincceker_23090/ (O. Kilincceker);
https://win.uantwerpen.be/~sdemey/ (S. Demeyer)
Orcid 0000-0001-7081-5385 (J. Kisaakye); 0000-0001-5996-4398 (O. Kilincceker); 0000-0002-4463-2945 (S. Demeyer)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
Mutation testing has been applied to various kinds of programming languages at the unit
level, the integration level, and the specification level [4, 5]. However, the use of mutation
testing for embedded software is a rather immature area of research and needs different methods
than conventional mutation testing. Indeed, modern embedded software is run on specific
hardware systems with respect to various constraints such as time and memory. In automotive
for instance, embedded systems can grow to 100 million lines of code running on 70 to 100
microprocessors [6]. At that scale, mutation testing at the code level becomes impractical:
we need to move to a higher abstraction level: model-based design. Indeed, to address the
scalability issue, one promising avenue of research called model-based mutation testing employs
abstract models rather than software programs [7, 8, 9, 10, 11]. The model-based mutation
testing methods mainly focus on conformance relations between the abstract models and their
corresponding software programs. These methods assume the presence of a correct abstract
model, generate test cases from the model and use mutation testing to select the best test cases.
MATLAB/Simulink, developed by The MathWorks, enables engineers to design, analyze, and
test embedded software by using model-based design. It permits testing at early-stage software
artifacts on simulations that can be automatically generated from Simulink models. To this
end, Simulink offers a solution to apply mutation testing on higher level design of embedded
software.
This paper aims to show feasibility of mutation testing on Simulink software artifacts. It
offers an automation framework to automate mutation testing to cope with testing of complex
embedded software systems at earlier phases to target unit level faults. Also, we introduce a
pilot study to conduct feasibility of the proposed framework. To the best of our knowledge, this
is the first work to uncover mutation testing on Simulink software artifacts based on the related
work provided in the next section.
The next section presents related work on mutation testing on Simulink models or its corre-
sponding artifacts including mutation operators. Section 3 introduces the proposed approach.
In section 4, we propose a pilot study to apply the proposed framework. Section 5 provides
discussion on open challenges including research and development opportunities. We conclude
our work in Section 6.
2. Related Work
Application of mutation testing on Simulink models or its software artifacts is an immature area
and there exist several challenges based on definition of mutation operators and automatization
of the processes.
Zhan and Clark proposed a method for search-based automatic test generation and measured
its adequacy using mutation testing for a Simulink model [12]. They used a targeted heuristic
search algorithm to capture a test that has the ability to kill mutants and compared their methods
with random testing. Their work confirmed that even for a limited set of mutants (30 mutants
were generated for a system of six blocks), mutation coverage can be used to assess the adequacy
of a test suite.
Brillout et al. used a bounded model checking method for test data generation aimed to
achieve high mutation coverage for Simulink models [13]. Their approach attempts to find a
Table 1
Comparison of Related Studies
Study Purpose Artefact Cons
[12] Test Generation Model Limited Mutation Operators
[13] Test Generation Model Cost of Model Checking
[14] Test Generation Model Manual Work
[16] Model-clone detection Model Semi-Automatic
[17] Mutation Testing Model -
Ours Mutation Testing C++ code Missing Evaluation
counterexample for demonstrating discrepancy between original and the mutated model. The
counterexamples are used to obtain test data. They plan the experimental evaluation as a future
work.
Hanh and Binh presented mutation operators for mutation testing of the Simulink model to
address the most common faults made by designers [14]. They divided fault classes for Simulink
models into six categories and deduced five categories of mutation operators for which they
informally defined different types of mutation operators. For a quadratic model pilot study,
they generated 59 mutants and wrote four sets of test cases. They achieved a 0,98 mutation
score. However, the mutation testing is applied manually that affects validity and reliability of
their results. As follow-up research, Hanh et al. also proposed a test data generation method
for mutation testing by using an artificial immune system for Simulink models [15].
Rather than mutation testing for Simulink software, Stephan et al. applied mutation testing
for Simulink models for detecting model-clones and defined component level mutation operators
[16]. They evaluated the mutation operators on a case study for multiple versions of three
Simulink projects. They separated results for each mutation operator in terms of mutation score
to measure their effectiveness. Their work was performed semi-automatically.
Besides manual and semi-automated works for mutation testing of Simulink models, Phil et
al. proposed an open-source toolset called SIMULTATE to automate the entire process including
injecting faults [17].
Based on the provided related work, we conclude that mutation testing of Simulink models
is feasible. However, there is not yet a consensus on an effective suite of mutation operators
and more experience reports are certainly welcome as well. More importantly, no study has
been conducted on the application of mutation testing to Simulink software artifacts. Table 1
compares related studies considering their purpose, artifacts applied on, and their disadvantages
compared with our proposal. To this end, the current work proposes an automation framework
to address this shortcoming.
3. Proposed Approach
We intend to construct an automation framework for mutation testing of Simulink models
integrated in state-of-the-art continuous integration (CI) and continuous delivery (CD) build
pipelines. The design of the framework centers around three steps: test design, mutant genera-
Figure 1: The proposed framework
tion, and test execution, respectively (see Figure 1).
In the test design step, unit level and model level test case will be written by using the
Simulink Test environment. This step is critical for addressing specific fault types that can occur
in the Simulink model. There needs to be correlation between mutants generated at the second
step to these types of faults and their corresponding test cases. Therefore, this step requires
expertise on designing test cases and mutation testing.
In the second step, the mutants of the Simulink model (written in the C++ programming
language) will be generated by utilizing an open-source mutation analysis tool. We envision the
use of Dextool to automatically generate mutants. Dextool employs specific mutation operators
for the C++ programming language. These operators are ABS (Absolute Value Insertion),
UOI (Unary Operator Insertion), LCR (Logical Connector Replacement), AOR (Arithmetic
operator replacement), and ROR (Relational Operator Replacement). We may consider additional
mutation operators, such as proposed by Parsai et al. [18] and Delgado-Pérez et al. [19].
Test scripts are executed automatically on the generated mutants in the third step. The
execution process will be carried on the Simulink Test environment. Based on these executions,
the framework is to output a test report that contains the overall mutation score and specific
mutation scores for mutation operators.
Figure 2: The control model of the robot system
The framework targets an automated build server. We are currently evaluating the appropriate
automation servers for which options are Azure® DevOps, CircleCI, Travis CI, or Jenkins with
GitHub repo.
Different from related work, the proposed framework is feasible considering its roadmap
to automate mutation testing on Simulink software. It also employs an open-source Dextool
integrated in continuous integration (CI) and continuous delivery (CD) build pipelines.
4. Pilot Study: Lego Robot System
Robotic systems are a common part of the manufacturing industry today. They automate
the applications to reduce costs and time for labor related processes [20]. MATLAB/Simulink
supports robot systems such as LEGO MINDSTORMS EV3 containing ARM9-based processors.
It also enables engineers to program and run-on LEGO MINDSTORMS EV3 robots [21].
The pilot study called the Lego robot system uses two colour sensors to follow a black line
and 1 gyro sensor to stop its motion, when placing a dark surface in front of the gyro sensor,
the robot stops. The line is created by placing white tiles with a black line on the floor. Its
controller is implemented in MATLAB/Simulink. The controller is run on a RaspberryPI CPU
which connects to the actuators and sensors using a BrickPI interface.
Figure 2 shows the control model of the robot, there are different sub models for the BrickPI
sensors, actuators, fault injection inputs and fault injection outputs. There are several Simulink
scopes that allow one to observe the progression of the different signals generated during the
operation of the robot. The Fault injection blocks allow one to set several properties of the fault
injection while the block labelled “Design your algorithm here” allows one to set the properties
of the Proportional, Integral, Derivative (PID) control of the robot.
Since the proposed framework applied on C++ code of the Simulink models, we automatically
generate C++ code of the control model of the robot system using Simulink Coder package.
Then, we employ the framework using Dextool for automatic mutant generation in the second
step. The already written unit tests in the first step will be executed on the generated mutants.
Dextool provides us desired metrics as being an output of the mutation testing procedure.
Thanks to the Simulink Test environment, all these steps run-on an automation framework.
5. Discussion
This section presents a discussion on possible threats to the validity of the current work, provides
open research challenges on mutation testing of Simulink models and its software artifacts, and
explains further studies within the scope.
5.1. Threats to the Validity
The following subsections discuss threats to internal and external validity. Also, it provides
possible mitigation solutions.
5.1.1. Internal Validity
To cope with complexity caused by redundant mutants, Simulink model is a good candidate
to apply mutation testing. Because Simulink code is lower level than the model level and
contains more structural information. However, the advances of mutation testing provide us
with necessary resources to tackle complexity issues of mutation testing, such as redundant
mutants. Mutation testing is currently state-of-the-practice rather than state-of-the-art [22, 1].
5.1.2. External Validity
The current work addresses unit level faults rather than model level or integration faults. How-
ever, we plan to integrate our framework on model level to address higher level faults. Moreover,
there is high correlation between model blocks, their integration, and their corresponding codes.
Within the scope of further experimentation, we plan to investigate their correlation to address
higher level faults by applying lower-level mutation testing.
5.2. Open Research Challenges and Further Works
There are various challenges and opportunities while applying mutation testing on Simulink
models are its corresponding C++ programs. They can be separated based on the artifacts
applied on.
On C++ programs, the mutation operators are mostly general-purpose and covering con-
ventional usage of the programming languages. The Simulink models utilized for embedded
systems are more specific and demanding definite mutation operators. For example, C++ on
Simulink is not case sensitive, while general purpose C++ is case sensitive. Readers may refer
[23] for more details between C++ and MATLAB. Therefore, the mutation operators addressing
case sensitivity become irrelevant and cause redundant mutants. Fault models for systems
modelled by Simulink are also specific, and thus corresponding tests become more explicit than
general-purpose C++ programs. To this end, the definition of explicit mutation operators and
preparation of tests addressing specific fault models become more challenging and provide us
many opportunities.
On Simulink models, there is not any consensus which mutation operators and how mutation
testing will be applied based on the provided related works. Except Phil et al. proposed an
open-source toolset called SIMULTATE [17], there are also missing comprehensive works on
automatization of the mutation testing. Already proposed works attempt to cover entire blocks
on Simulink models by using generic mutation operators. However, tests aim to cover each
specified block. Therefore, the definition of mutation operators for different types of model
blocks needs to be more specific. Based on these challenges for mutation testing on Simulink
models, there are many research and development opportunities.
Since we address only automatization of mutation testing on Simulink code integrated in
state-of-the-art continuous integration (CI) and continuous delivery (CD) build pipelines, we
keep the above-mentioned challenges for further studies. The utilized open-source Dextool
provides us with a selection of desired mutation operators to experiment their effectiveness on
the case study.
6. Conclusion
In this paper we argued the need for model-based mutation testing, in the context of MAT-
LAB/Simulink. Based on an overview of related work, we conclude that such a tool is feasible,
however that there is not yet a consensus on an effective suite of mutation operators. No study
has been proposed on the application of mutation testing to Simulink software artifacts. This
work aims to fill this gap. We therefore propose a proof-of-concept tool (currently in the design
phase) that would integrate Dextool, MATLAB/Simulink and a continuous integration build
server. We intend to validate the tool on a line follower robot, implemented in Lego Mindstorms
controlled by a MATLAB/Simulink model. Moreover, we discuss open research challenges and
further studies including possible threats to the validity of the current work.
Acknowledgments
The authors express their gratitude to anonymous reviewer for their valuable comments and
suggestions on earlier versions of the current paper. This work is supported by the Flanders
Innovation and Entrepreneurship under grant number HBC.2021.0010 entitled EFFECTS.
References
[1] A. Parsai, S. Demeyer, Comparing mutation coverage against branch coverage in an
industrial setting, International Journal on Software Tools for Technology Transfer 22
(2020) 365–388.
[2] R. G. Hamlet, Testing programs with the aid of a compiler, IEEE transactions on software
engineering (1977) 279–290.
[3] R. A. DeMillo, R. J. Lipton, F. G. Sayward, Hints on test data selection: Help for the
practicing programmer, Computer 11 (1978) 34–41.
[4] Y. Jia, M. Harman, An analysis and survey of the development of mutation testing, IEEE
transactions on software engineering 37 (2010) 649–678.
[5] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, M. Harman, Mutation testing
advances: an analysis and survey, in: Advances in Computers, volume 112, Elsevier, 2019,
pp. 275–378.
[6] R. N. Charette, This car runs on code, IEEE spectrum 46 (2009) 3.
[7] W. Krenn, R. Schlick, S. Tiran, B. Aichernig, E. Jobstl, H. Brandl, Momut:: Uml model-based
mutation testing for uml, in: 2015 IEEE 8th International Conference on Software Testing,
Verification and Validation (ICST), IEEE, 2015, pp. 1–8.
[8] B. K. Aichernig, J. Auer, E. Jöbstl, R. Korošec, W. Krenn, R. Schlick, B. V. Schmidt, Model-
based mutation testing of an industrial measurement device, in: International Conference
on Tests and Proofs, Springer, 2014, pp. 1–19.
[9] F. Belli, C. J. Budnik, A. Hollmann, T. Tuglular, W. E. Wong, Model-based mutation
testing—approach and case studies, Science of Computer Programming 120 (2016) 25–48.
[10] O. Kilincceker, A. Silistre, F. Belli, M. Challenger, Model-based ideal testing of gui programs–
approach and case studies, IEEE Access 9 (2021) 68966–68984.
[11] O. Kilincceker, E. Turk, F. Belli, M. Challenger, Model-based ideal testing of hardware
description language (hdl) programs, Software and Systems Modeling (2021) 1–32.
[12] Y. Zhan, J. A. Clark, Search-based mutation testing for simulink models, in: Proceedings of
the 7th annual conference on Genetic and evolutionary computation, 2005, pp. 1061–1068.
[13] A. Brillout, N. He, M. Mazzucchi, D. Kroening, M. Purandare, P. Rümmer, G. Weissenbacher,
Mutation-based test case generation for simulink models, in: International Symposium on
Formal Methods for Components and Objects, Springer, 2009, pp. 208–227.
[14] N. T. Binh, et al., Mutation operators for simulink models, in: 2012 Fourth International
Conference on Knowledge and Systems Engineering, IEEE, 2012, pp. 54–59.
[15] L. T. M. Hanh, N. T. Binh, K. T. Tung, A novel test data generation approach based upon
mutation testing by using artificial immune system for simulink models, in: Knowledge
and Systems Engineering, Springer, 2015, pp. 169–181.
[16] M. Stephan, M. H. Alalfi, J. R. Cordy, Towards a taxonomy for simulink model mutations,
in: 2014 IEEE Seventh International Conference on Software Testing, Verification and
Validation Workshops, IEEE, 2014, pp. 206–215.
[17] I. Pill, I. Rubil, F. Wotawa, M. Nica, Simultate: A toolset for fault injection and mutation
testing of simulink models, in: 2016 IEEE Ninth International Conference on Software
Testing, Verification and Validation Workshops (ICSTW), IEEE, 2016, pp. 168–173.
[18] A. Parsai, S. Demeyer, S. De Busser, C++ 11/14 mutation operators based on common fault
patterns, in: IFIP International Conference on Testing Software and Systems, Springer,
2018, pp. 102–118.
[19] P. Delgado-Pérez, I. Medina-Bulo, F. Palomo-Lozano, A. García-Domínguez, J. J. Domínguez-
Jiménez, Assessment of class mutation operators for c++ with the mucpp mutation system,
Information and Software Technology 81 (2017) 169–184.
[20] www.robots.com, Three types of robotic systems, 2021. URL: https://www.robots.com/
articles/three-types-of-robotic-systems.
[21] E. Schoofs, J. Kisaakye, B. Karaduman, M. Challenger, Software agent-based multi-robot
development: A case study, in: 2021 10th Mediterranean Conference on Embedded
Computing (MECO), IEEE, 2021, pp. 1–8.
[22] H. Coles, T. Laurent, C. Henard, M. Papadakis, A. Ventresque, Pit: a practical mutation
testing tool for java, in: Proceedings of the 25th international symposium on software
testing and analysis, 2016, pp. 449–452.
[23] Mathworks, Comparison of matlab and other oo languages, 2021. URL: https://nl.
mathworks.com/help/matlab/matlab_oop/matlab-vs-other-oo-languages.html.