=Paper=
{{Paper
|id=Vol-2245/modevva_paper_2
|storemode=property
|title=Generation of Test Strategies for Model-based Functional Safety Testing using an Artifact-centric Approach
|pdfUrl=https://ceur-ws.org/Vol-2245/modevva_paper_2.pdf
|volume=Vol-2245
|authors=Bert Van Acker,Joachim Denil,Paul De Meulenaere,Bjorn Aelvoet,Dries Mahieu,Jan van den Oudenhoven
|dblpUrl=https://dblp.org/rec/conf/models/AckerDMAMO18
}}
==Generation of Test Strategies for Model-based Functional Safety Testing using an Artifact-centric Approach==
https://ceur-ws.org/Vol-2245/modevva_paper_2.pdf
Generation of Test Strategies for Model-based Functional
Safety Testing using an Artifact-Centric Approach
Bert Van Acker, Joachim Denil, Paul De Meulenaere
CoSys-lab
University of Antwerp, Belgium
Flanders Make, Belgium
Bert.VanAcker@uantwerp.be, Joachim.Denil@uantwerp.be, Paul.Demeulenaere@uantwerp.be
Bjorn Aelvoet, Dries Mahieu, Jan van den Oudenhoven
DANA BELGIUM NV
bjorn.aelvoet@dana.com, dries.mahieu@dana.com, jan.vandenoudenhoven@dana.com
between software and physical elements[Lee08], it is
also caused by the safety aspects of such systems.
Abstract Safety engineering is a cross-cutting concern that is
taken into account throughout the complete life-cycle.
ISO/DIS 26262, an automotive func- Safety engineering aims to show that the required
tional safety standard, provides strin- system functionality is safe and reliable[Bellotti10].
gent requirements and processes for a This way, the quality and reliability of safety critical
’safety-oriented’ software lifecycle and systems are highly dependent on both the implemen-
in particular on the verification and val- tation and proper system validation and verification.
idation part. These strict and activity- To cope with this complexity, engineers can adapt
based safety processes impose some im- model-based engineering principles, enabling complex
portant drawbacks, especially with the system analysis via system simulations. Simulations
increasing complexity of software inten- allow for a preliminary validation in very early stages
sive safety critical systems. In this of the development process, this reduces the risk of
paper we report on a methodology high redesign cost by detecting premature errors.
for guiding the Model-based Functional Moreover, safety critical systems must adhere to
Safety testing by generating valid test functional safety norms like IEC 61508[IEC61508]
strategy models. We explicitly model and ISO/DIS 26262[ISO/DIS26262]. These standards
test artifacts and process rules, which pose stringent requirements for development of safety
allows to automatically generate valid critical systems and also on the testing processes.
and optimized test strategies for Model- Adhering to these standards can be difficult because
based Functional Safety testing. A well- the defined activity-centric processes are rigid and
known advanced driver assistance sys- some margin for appreciation can exist, so imple-
tem, the adaptive cruise control, is used menting an instantiation of the requirements outlined
to demonstrate the proposed methodol- in ISO/DIS 26262 within a Model-Based Design
ogy. requires special consideration[Conrad12]. Moreover,
the verification and validation (V&V) processes,
1 Introduction which contain not only testing activities but also
e.g. (static) analysis, inspections and reviews, etc.,
Development of software intensive safety critical are preferably application-specific meaning that the
systems, such as found for example in the avionics, instantiation of the V&V process is fitted for the
automotive, maritime, and energy domains, is becom- application in development, which can reduce the
ing increasingly complex. This increasing complexity V&V overhead.
is not only caused by the synergistically interaction
violation. A clear example of a safety function is a safe
In our approach, we propose to focus on the standstill function in road vehicles which will monitor
artifacts instead of the fixed process flows outlined in the vehicles movement during standstill and will acti-
the ISO/DIS 26262. Artifacts are the products used vate the park brake if an unintended movement occurs.
by and/or generated from process activities and the
resulting artifact-centric process models are defined This guidance is activity-based, meaning that a
as acceptable states of the process without enforc- chain of activities is provided and the control flow
ing any specific execution flow[Baresi16]. Process is well defined and fixed. This works very good for
rules are used to formally capture the constraints defining the normal execution flow and the manage-
of a valid process execution flow, enabling us to ment of foreseen exceptions during execution of the
(semi-)automatically generate valid, optimized and processes but when unforeseen situations occur, the
customized test strategy processes for Model-based correct execution within the process definition can-
Functional Safety testing. Process rules can originate not be checked[Baresi16]. Artifact-centric processes
from various sources such as company-specific process could provide a good solution to handle these unfore-
knowledge or safety-specific knowledge captured in seen situations as it provides a process definition in
the functional safety standard. By applying these terms of acceptable process states and not enforcing
process rules at different steps in the workflow, we a specific execution flow. [Yongchareon18] states that
can truly make the test strategies application- and artifact-centric process modeling has been evidenced
company-specific, facilitating the usage of the test with higher flexibility over traditional activity-centric
strategy processes and further reducing the V&V process modeling and they used it to improve inter-
overhead. organizational business cooperation. [Kuhrmann14]
also states that by focusing on the artifacts, which
The rest of the paper is organised as follows : Sec- precisely define the desired outcomes, rather than on
tion 2 presents some essential background and related specific methods, the processes are less generic and
work on functional safety testing and the used artifact- more fitted for the organization.
centric approach. Section 3 discusses the basic con-
cepts of the presented solution. Section 4 implements 3 Approach
a case study and finally, Section 5 concludes the paper.
This section introduces the proposed methodology of
our approach, which is shown conceptually in Figure
2 Background and related work 1.
Safety is one of the key issues in future road vehicle The result of our approach is a (semi-)automatically
development[Bellotti10] and despite the increasing generated valid and optimized test strategy process
complexity, caused by new complex driver assistance compliant to the ISO/DIS 26262 functional safety
systems with e.g. accident prediction and avoidance standard. We start from an integration model(top-left
capabilities and the associated complicated processes, in Figure 1), where a set of components are connected
car manufacturers have the basic obligation to only together to form a particular safety function. Each
put safe products on the market. In safety critical of these components and their integration need to
systems, human safety depends upon the correct be tested compliant the functional safety standard.
operation of those systems. This raises the need for These components can originate from different sources
development methods and processes that could lead to such as an internal software department or external
provable correct systems[Arc17]. Therefore, car mak- suppliers and can already been tested partially or
ers as well as suppliers adhere to a functional safety completely.
standard, namely the ISO/DIS 26262. This standard
gives stringent requirements for a ’safety-oriented’ This knowledge about the level of test comple-
design and in particular on the verification and vali- tion is captured for each component 1 within the
dation part. A high-level guidance to design proper artifact-view of the component. The corresponding
safety functions is provided in the guidelines, together artifact-view of the components are shown in the
with processes to verify and validate these safety top-middle of Figure 1 as parallelograms with four
functions and prove their adherence to the standard. squares at the bottom. These squares depict the
The aforementioned safety functions are functions different integration levels of the system under test
implemented in a control system to prevent violation (SUT) within the model-based testing of embedded
of the safety goals. These safety functions will assure systems, namely Model-in-the-Loop (MiL), Software-
safe operation of the system and will put the system 1 With ’component’, we intend to denote the components of
in a safe state in case of an inevitable safety goal the integrated system and the integrated system itself
Figure 1: Overview method
in-the-Loop(SiL), Processor-in-the-Loop(PiL) and selecting the proper V&V activity test methods and
Hardware-in-the-Loop(HiL)[Zander11]. Important applying the defined process rules, we automati-
remark with these integration levels is that the cally generate the valid, customized and optimized
’in-the-Loop’ part of the definition is not always model-based test strategy process for functional safety.
present in the testing of components/units as they
can be tested in open-loop setup. We however adhere In the following paragraphs we first look at a way
to this terminology in this paper because they still to formally capture the extra information about the
denote the right abstraction level of testing and their test level. Afterwards, we look at the process rules
corresponding testing methods. The crosses inside the and lastly, we discuss the needed transformations.
squares represent the test status where the presence
of the cross indicates that the tests are completed at 3.1 Artifact-view model
the corresponding integration level.
As stated before, the extra information about the
Next, we define process rules which will constrain level of model-based test completion needs to be cap-
the valid control flow of the model-based test activities tured for each of the components of the implementa-
and together with the artifact-view of each compo- tion model. We use an artifact-based approach where
nent, we automatically generate possible test strategy we explicitly model this extra meta-information in an
processes. Within these generated processes, each artifact-view model. This artifact-view contains the
model-based test activity at a particular integration necessary model-based test activities and their corre-
level can further be decomposed in V&V activities, sponding status in relation with the artifacts. The con-
e.g. functional safety behavior testing on MiL. For trol flow between these activities is not defined within
these V&V activities one or more ISO/DIS 26262 this artifact-view. The meta-model of the artifact-view
compliant testing method(s) needs to be selected. model and its dependencies is shown in Figure 2.
The selection or customization can influence other The meta-model contains two basic classes:
parts of the process and the rules to define these
dependencies are also explicitly modeled using the • Artifact class defines an artifact, which repre-
aforementioned process rules. The process rules sents an implementation of the component, e.g.
are thus used at different stages of the workflow the model or source code representing the func-
but are conceptually equal, so a generic modeling tionality of the component. These serve as input
mechanism is used to define these process rules. After for and output from the activities.
Figure 3: Graphical example of a process rule
where activity 13 and activity 14 induce a particular
rule to activity 24.
Following rules can be applied:
Figure 2: Artifact-view meta model and dependencies • Sequential execution of activities
• Activity class defines an activity. The status is • Parallel execution of activities
a key element captured within this artifact-view.
• Activity not feasible within valid process
In the scope of functional safety testing, where • Preferred V&V method for activity
the possible V&V test methods are imposed by the
ISO/DIS 26262 standard, the method attribute in • V&V method selection forces V&V method for
the activity object enables the customization of the other activity
test strategy by selecting one or more available V&V
test methods. The process rules on the other hand Note that these set of rules can easily be extended
will have influence on the activities, e.g. they can with extra process rules. An example of a more ad-
alter the available V&V test methods or omit an vanced process rule, originating from both the safety
activity depending on other activities. The process standard and company-specific knowledge is defined as
rules will also define the control flow between the follows: if the functional behavior of a component is
activities, deemed necessary to generate a valid test tested via a requirements-based test method on model
strategy process. Note that an artifact-view can level(MiL), we force the V&V test method on soft-
contain links to other artifact-views, indicating that ware level(SiL), namely a back-to-back test method
the artifact-view corresponds to an integration model combined with an interface test method.
of different components. Up till now, we used a text-based model to define
the process rules. This could be improved by defining a
In our approach, the artifact-view model is a text- graphical domain-specific language (DSL), which eases
based model defined with an extensive mark-up lan- the modeling of these process rules.
guage (XML). This enables the model to be human-
and machine readable, which is deemed necessary to 3.3 Transformations
process this model in the subsequent steps of the work- With the artifact-views and the process rules defined,
flow. we can generate a set of valid test strategy processes.
This transformation extracts the meta-information
3.2 Process rules of all artifact-views and generates a process model
where the appropriate process rules are taken into ac-
The key element of our proposed approach is the ex-
count. The generated process model can either be a
plicit modeling of process rules which enables the op-
directed graph or a Causal Block Diagram representing
timization and customization of the test strategy pro-
a (executable) process model. This process model can
cesses. These rules can be applied at different stages of
further be customized by selecting the proper V&V
the workflow, depending on the rules and their impact.
test methods and again applying the specified process
As mentioned before, rules can e.g. originate from re-
rules. This will be further explained by means of a
quirements posed by standards or company-specific de-
practical example in Section 4.
cisions. Important benefit of explicitly modeling these
process rules is that all process rules or decisions are
formally captured and can be used as well-documented
4 Case Study
evidence to prove process compliance to certain stan- In this section, we use a safety critical adaptive cruise
dards. control system as academic use case to illustrate
Figure 3 is a graphical example of a process rule the proposed concepts of the previous section. The
Figure 5: Safety chain schematic artifact-view
Figure 4: SafeDistance safety function
adaptive cruise control system is an advanced driver
assistance system (ADAS) which is already widely
available in commercial road vehicles but interesting
as system under study because it is one of the
precursors of fully autonomous vehicles[Nardi17].
For this use case, we will decrease the complexity
by focusing on one safety function defined in the
implementation of the adaptive cruise control, namely
the SafeDistance function. The safety chain for this
SafeDistance safety function is shown in Figure 4.
This safety chain is decomposed in four compo-
nents, each on a different level of abstraction. More
specifically, each component is unit tested up till a
certain test integration level. This knowledge is for-
mally captured in the corresponding artifact-view of Figure 6: Model-based functional safety test process
each component. The artifact-view of the complete
performed after MiL, (ii)Pil and Hil need to be se-
SafeDistance safety function is shown graphically in
quentially performed after SiL, (iii)Pil and HiL can be
Figure 5. This defines that the AccMonitor compo-
performed in parallel and (iiii) integration tests need
nent, leftmost artifact-view in Figure 5 and highlighted
to be sequential performed after the completion of the
in the red box in Figure 4, is unit tested up till the soft-
unit tests of all integrated components.
ware integration level. To complete the unit test for
this component, the tests on the processor and hard- With the artifact-views and the process rules
ware integration level need to be performed. From this defined, a valid test strategy process model can
graphical representation, the artifacts are not explic- automatically be generated. This text-to-model
itly defined2 as they are implicit present in the activity transformation generates a directed graph, as shown
names, namely: in Figure 6, where the test activities are depicted as
nodes and the control flow as links.
• Model-in-the-Loop: Input artifact is a model
of the AccMonitor In this use case, we not only generate a directed
• Software-in-the-Loop: Input artifact is source graph, we also generate a process model in a Causal
code of the AccMonitor Block Diagram (CBD) formalism in Simulink c , which
allows us to visually select the proper V&V test meth-
• Processor/Hardware-in-the-Loop: Input ar- ods for the test activities, when necessary, to truly
tifact is production code of the AccMonitor customize the test strategy process. An example of a
test activity in CBD formalism is shown in Figure 7,
Besides the necessary knowledge about the test
where the selection of the V&V test methods, if appli-
completion, we also define general process rules for
cable, is present for the sub-activities. The available
a valid execution flow of the model-based test activ-
V&V test methods and the possible test case deriva-
ities. We define that (i)SiL needs to be sequentially
tion methods are compliant to the ISO/DIS 26262 as
2 Artifacts are also not explicitly present in the processes shown by the SpecifiedBy relation between the tables,
Figure 7: Configurable functional safety test activity
and ISO/DIS 26262 compliance relation
originating from ISO/DIS 26262 - part 6, and the cus-
tomizable CBD block.
At this level, process rules are defined to capture the
dependencies between the different test activities and
their corresponding V&V test methods. The following
list is a subset of the outcome of the applied process
rules for this use case:
Figure 8: Generated valid, optimized and customized
test strategy
• Resource usage test not feasible on MiL
5 Conclusion and future work
• Back-to-back test not feasible on MiL
This paper presents a methodology to facilitate the
• Requirements based test on MiL forces require- model-based validation and verification of safety
ments based test and boundary values test on SiL critical systems by (semi-)automatically generating
valid, optimized test strategy processes compliant
• PiL not feasible for unit testing to the ISO/DIS 26262 functional safety standard
using an artifact-centric approach. We applied this
• HiL not feasible for unit testing methodology to a well-known advanced driver assis-
tance system, the adaptive cruise control, extended
with the needed safety functions. This case study is
After selecting the proper V&V test methods3 small-scale and yet complex enough to be suited for
and applying these process rules, a valid, optimized future research.
and customized test strategy model is automatically
generated.
In the future, we plan to extend the proposed
methodology by introducing one or more design space
An example of a generated test strategy model is exploration (DSE)algorithms to further optimize the
shown in Figure 8. In this generated test strategy generation of valid test strategy processes. By taking
process,the proper V&V test methods are selected the resource constraints, such as shared real-time
and the above mentioned process rules are applied. To hardware platforms or human resources, into account,
increase the readability and usability of the generated both a valid and optimized test strategy process
test strategy model, we grouped the sub-activities of and an optimized test scheduling can be generated,
each test activity at the particular integration level. which is beneficially for the overall verification and
validation processes.
Second, we want to extend the usage of the
3 The applied process rules have also influence on the avail- artifact-view by tightly coupling this artifact-view to
able/valid V&V test methods the model information or meta-knowledge[Sirin14],
enabling the use in design processes other than safety- [IEC61508] IEC 61508 ed. 2.0. International Elec-
related automotive development. More specifically, trotechnical Commission, IEC;, 2009.
we will include this information of the artifact-view
within the validity frame[Denil17][Klikovits17] encap- [ISO/DIS26262] ISO/DIS 26262 (all parts), road ve-
sulated with each component. This will increase the hicles functional safety. International Orga-
(re-)usability of the components within the design nization for Standardization.
processes. [Klikovits17] Klikovits, S., Denil, J., Muzy, A., &
Salay, R. Modeling frames MoDeVVa Work-
Lastly, we will compare our proposed methodology shop Proceedings, 2017.
against more traditional activity-based processes to
empirically assess the usability, correctness and scala- [Kuhrmann14] Kuhrmann, M., & Beecham, S.
bility of our methodology. Artifact-Based Software Process Improve-
ment and Management : A Method Pro-
6 Acknowledgments posal. 2018.
This work has been carried out within the framework [Lee08] E. Lee et al., Cyber physical systems: De-
of Flanders Make’s ICON project ’EnPower’ (TCO op- sign challenges, Object Oriented Real-Time
timal system design for Energy and Power storage in Distributed Computing (ISORC), 2008 11th
dynamic load applications, HBC.2016.0462) funded by IEEE International Symposium on. IEEE,
the agency Flanders Innovation & Entrepreneurship pp. 363369, 2008.
(VLAIO) and Flanders Make. Flanders Make is the
[Nardi17] Nardi, A., & Armato, A. Functional safety
Flemish strategic research centre for the manufactur-
methodologies for automotive applications.
ing industry.
IEEE/ACM International Conference on
Computer-Aided Design, Digest of Techni-
References cal Papers, ICCAD, 2017Novem, 970975.
https://doi.org/10.1109/ICCAD.2017.8203886,
[Arc17] Arcaini, P., Gargantini, A., & Riccobene,
2017.
E. Rigorous development process of a
safety-critical system: from ASM models to [Sirin14] G. Sirin, C. J. Paredis, B. Yannou, E.
Java code. International Journal on Soft- Coatanea, and E. Landel, A Model Identity
ware Tools for Technology Transfer, 19(2), Card to Support Simulation Model Develop-
247269. https://doi.org/10.1007/s10009- ment Process in a Collaborative Multidisci-
015-0394-x, 2017. plinary Design Environment, 2014.
[Baresi16] Baresi, L., Meroni, G., & Plebani, P. [Yongchareon18] Yongchareon, S., Jian, Y., &
On Handling Business Process Anomalies Zhao, X. A View Framework for Mod-
through Artifact-based Modeling, 2016. eling and Change Validation of Artifact-
Centric Inter-Organizational Business
[Bellotti10] Bellotti, M., & Mariani, R. How future Processes, Information Systems (April).
automotive functional safety requirements https://doi.org/10.1016/j.is.2014.07.004,
will impact microprocessors design. Micro- 2018.
electronics Reliability, 50(911), 13201326.
https://doi.org/10.1016/j.microrel.2010.07.041, [Zander11] Zander, J., Schieferdecker, I., & Moster-
2010. man, P. A Taxonomy of Model-Based
Testing for Embedded Systems from
[Conrad12] Conrad, M. Verification and Validation Multiple Industry Domains. Model-
According to ISO 26262 : A Workflow to Based Testing for Embedded Systems, 122.
Facilitate the Development of High-Integrity https://doi.org/doi:10.1201/b11321-2, 2011.
Software. roc. ERTS 2012 Embedded Real
Time Software Ans Systems., 2012.
[Denil17] J. Denil, S. Klikovits, P. J. Mosterman, A.
Vallecillo, and H. Vangheluwe, The exper-
iment model and validity frame in M&S,
Proceedings of the Symposium on Theory of
Modelling and Simulation, 2017.