Generation of Test Strategies for Model-based Functional Safety Testing using an Artifact-Centric Approach Bert Van Acker, Joachim Denil, Paul De Meulenaere CoSys-lab University of Antwerp, Belgium Flanders Make, Belgium Bert.VanAcker@uantwerp.be, Joachim.Denil@uantwerp.be, Paul.Demeulenaere@uantwerp.be Bjorn Aelvoet, Dries Mahieu, Jan van den Oudenhoven DANA BELGIUM NV bjorn.aelvoet@dana.com, dries.mahieu@dana.com, jan.vandenoudenhoven@dana.com between software and physical elements[Lee08], it is also caused by the safety aspects of such systems. Abstract Safety engineering is a cross-cutting concern that is taken into account throughout the complete life-cycle. ISO/DIS 26262, an automotive func- Safety engineering aims to show that the required tional safety standard, provides strin- system functionality is safe and reliable[Bellotti10]. gent requirements and processes for a This way, the quality and reliability of safety critical ’safety-oriented’ software lifecycle and systems are highly dependent on both the implemen- in particular on the verification and val- tation and proper system validation and verification. idation part. These strict and activity- To cope with this complexity, engineers can adapt based safety processes impose some im- model-based engineering principles, enabling complex portant drawbacks, especially with the system analysis via system simulations. Simulations increasing complexity of software inten- allow for a preliminary validation in very early stages sive safety critical systems. In this of the development process, this reduces the risk of paper we report on a methodology high redesign cost by detecting premature errors. for guiding the Model-based Functional Moreover, safety critical systems must adhere to Safety testing by generating valid test functional safety norms like IEC 61508[IEC61508] strategy models. We explicitly model and ISO/DIS 26262[ISO/DIS26262]. These standards test artifacts and process rules, which pose stringent requirements for development of safety allows to automatically generate valid critical systems and also on the testing processes. and optimized test strategies for Model- Adhering to these standards can be difficult because based Functional Safety testing. A well- the defined activity-centric processes are rigid and known advanced driver assistance sys- some margin for appreciation can exist, so imple- tem, the adaptive cruise control, is used menting an instantiation of the requirements outlined to demonstrate the proposed methodol- in ISO/DIS 26262 within a Model-Based Design ogy. requires special consideration[Conrad12]. Moreover, the verification and validation (V&V) processes, 1 Introduction which contain not only testing activities but also e.g. (static) analysis, inspections and reviews, etc., Development of software intensive safety critical are preferably application-specific meaning that the systems, such as found for example in the avionics, instantiation of the V&V process is fitted for the automotive, maritime, and energy domains, is becom- application in development, which can reduce the ing increasingly complex. This increasing complexity V&V overhead. is not only caused by the synergistically interaction violation. A clear example of a safety function is a safe In our approach, we propose to focus on the standstill function in road vehicles which will monitor artifacts instead of the fixed process flows outlined in the vehicles movement during standstill and will acti- the ISO/DIS 26262. Artifacts are the products used vate the park brake if an unintended movement occurs. by and/or generated from process activities and the resulting artifact-centric process models are defined This guidance is activity-based, meaning that a as acceptable states of the process without enforc- chain of activities is provided and the control flow ing any specific execution flow[Baresi16]. Process is well defined and fixed. This works very good for rules are used to formally capture the constraints defining the normal execution flow and the manage- of a valid process execution flow, enabling us to ment of foreseen exceptions during execution of the (semi-)automatically generate valid, optimized and processes but when unforeseen situations occur, the customized test strategy processes for Model-based correct execution within the process definition can- Functional Safety testing. Process rules can originate not be checked[Baresi16]. Artifact-centric processes from various sources such as company-specific process could provide a good solution to handle these unfore- knowledge or safety-specific knowledge captured in seen situations as it provides a process definition in the functional safety standard. By applying these terms of acceptable process states and not enforcing process rules at different steps in the workflow, we a specific execution flow. [Yongchareon18] states that can truly make the test strategies application- and artifact-centric process modeling has been evidenced company-specific, facilitating the usage of the test with higher flexibility over traditional activity-centric strategy processes and further reducing the V&V process modeling and they used it to improve inter- overhead. organizational business cooperation. [Kuhrmann14] also states that by focusing on the artifacts, which The rest of the paper is organised as follows : Sec- precisely define the desired outcomes, rather than on tion 2 presents some essential background and related specific methods, the processes are less generic and work on functional safety testing and the used artifact- more fitted for the organization. centric approach. Section 3 discusses the basic con- cepts of the presented solution. Section 4 implements 3 Approach a case study and finally, Section 5 concludes the paper. This section introduces the proposed methodology of our approach, which is shown conceptually in Figure 2 Background and related work 1. Safety is one of the key issues in future road vehicle The result of our approach is a (semi-)automatically development[Bellotti10] and despite the increasing generated valid and optimized test strategy process complexity, caused by new complex driver assistance compliant to the ISO/DIS 26262 functional safety systems with e.g. accident prediction and avoidance standard. We start from an integration model(top-left capabilities and the associated complicated processes, in Figure 1), where a set of components are connected car manufacturers have the basic obligation to only together to form a particular safety function. Each put safe products on the market. In safety critical of these components and their integration need to systems, human safety depends upon the correct be tested compliant the functional safety standard. operation of those systems. This raises the need for These components can originate from different sources development methods and processes that could lead to such as an internal software department or external provable correct systems[Arc17]. Therefore, car mak- suppliers and can already been tested partially or ers as well as suppliers adhere to a functional safety completely. standard, namely the ISO/DIS 26262. This standard gives stringent requirements for a ’safety-oriented’ This knowledge about the level of test comple- design and in particular on the verification and vali- tion is captured for each component 1 within the dation part. A high-level guidance to design proper artifact-view of the component. The corresponding safety functions is provided in the guidelines, together artifact-view of the components are shown in the with processes to verify and validate these safety top-middle of Figure 1 as parallelograms with four functions and prove their adherence to the standard. squares at the bottom. These squares depict the The aforementioned safety functions are functions different integration levels of the system under test implemented in a control system to prevent violation (SUT) within the model-based testing of embedded of the safety goals. These safety functions will assure systems, namely Model-in-the-Loop (MiL), Software- safe operation of the system and will put the system 1 With ’component’, we intend to denote the components of in a safe state in case of an inevitable safety goal the integrated system and the integrated system itself Figure 1: Overview method in-the-Loop(SiL), Processor-in-the-Loop(PiL) and selecting the proper V&V activity test methods and Hardware-in-the-Loop(HiL)[Zander11]. Important applying the defined process rules, we automati- remark with these integration levels is that the cally generate the valid, customized and optimized ’in-the-Loop’ part of the definition is not always model-based test strategy process for functional safety. present in the testing of components/units as they can be tested in open-loop setup. We however adhere In the following paragraphs we first look at a way to this terminology in this paper because they still to formally capture the extra information about the denote the right abstraction level of testing and their test level. Afterwards, we look at the process rules corresponding testing methods. The crosses inside the and lastly, we discuss the needed transformations. squares represent the test status where the presence of the cross indicates that the tests are completed at 3.1 Artifact-view model the corresponding integration level. As stated before, the extra information about the Next, we define process rules which will constrain level of model-based test completion needs to be cap- the valid control flow of the model-based test activities tured for each of the components of the implementa- and together with the artifact-view of each compo- tion model. We use an artifact-based approach where nent, we automatically generate possible test strategy we explicitly model this extra meta-information in an processes. Within these generated processes, each artifact-view model. This artifact-view contains the model-based test activity at a particular integration necessary model-based test activities and their corre- level can further be decomposed in V&V activities, sponding status in relation with the artifacts. The con- e.g. functional safety behavior testing on MiL. For trol flow between these activities is not defined within these V&V activities one or more ISO/DIS 26262 this artifact-view. The meta-model of the artifact-view compliant testing method(s) needs to be selected. model and its dependencies is shown in Figure 2. The selection or customization can influence other The meta-model contains two basic classes: parts of the process and the rules to define these dependencies are also explicitly modeled using the • Artifact class defines an artifact, which repre- aforementioned process rules. The process rules sents an implementation of the component, e.g. are thus used at different stages of the workflow the model or source code representing the func- but are conceptually equal, so a generic modeling tionality of the component. These serve as input mechanism is used to define these process rules. After for and output from the activities. Figure 3: Graphical example of a process rule where activity 13 and activity 14 induce a particular rule to activity 24. Following rules can be applied: Figure 2: Artifact-view meta model and dependencies • Sequential execution of activities • Activity class defines an activity. The status is • Parallel execution of activities a key element captured within this artifact-view. • Activity not feasible within valid process In the scope of functional safety testing, where • Preferred V&V method for activity the possible V&V test methods are imposed by the ISO/DIS 26262 standard, the method attribute in • V&V method selection forces V&V method for the activity object enables the customization of the other activity test strategy by selecting one or more available V&V test methods. The process rules on the other hand Note that these set of rules can easily be extended will have influence on the activities, e.g. they can with extra process rules. An example of a more ad- alter the available V&V test methods or omit an vanced process rule, originating from both the safety activity depending on other activities. The process standard and company-specific knowledge is defined as rules will also define the control flow between the follows: if the functional behavior of a component is activities, deemed necessary to generate a valid test tested via a requirements-based test method on model strategy process. Note that an artifact-view can level(MiL), we force the V&V test method on soft- contain links to other artifact-views, indicating that ware level(SiL), namely a back-to-back test method the artifact-view corresponds to an integration model combined with an interface test method. of different components. Up till now, we used a text-based model to define the process rules. This could be improved by defining a In our approach, the artifact-view model is a text- graphical domain-specific language (DSL), which eases based model defined with an extensive mark-up lan- the modeling of these process rules. guage (XML). This enables the model to be human- and machine readable, which is deemed necessary to 3.3 Transformations process this model in the subsequent steps of the work- With the artifact-views and the process rules defined, flow. we can generate a set of valid test strategy processes. This transformation extracts the meta-information 3.2 Process rules of all artifact-views and generates a process model where the appropriate process rules are taken into ac- The key element of our proposed approach is the ex- count. The generated process model can either be a plicit modeling of process rules which enables the op- directed graph or a Causal Block Diagram representing timization and customization of the test strategy pro- a (executable) process model. This process model can cesses. These rules can be applied at different stages of further be customized by selecting the proper V&V the workflow, depending on the rules and their impact. test methods and again applying the specified process As mentioned before, rules can e.g. originate from re- rules. This will be further explained by means of a quirements posed by standards or company-specific de- practical example in Section 4. cisions. Important benefit of explicitly modeling these process rules is that all process rules or decisions are formally captured and can be used as well-documented 4 Case Study evidence to prove process compliance to certain stan- In this section, we use a safety critical adaptive cruise dards. control system as academic use case to illustrate Figure 3 is a graphical example of a process rule the proposed concepts of the previous section. The Figure 5: Safety chain schematic artifact-view Figure 4: SafeDistance safety function adaptive cruise control system is an advanced driver assistance system (ADAS) which is already widely available in commercial road vehicles but interesting as system under study because it is one of the precursors of fully autonomous vehicles[Nardi17]. For this use case, we will decrease the complexity by focusing on one safety function defined in the implementation of the adaptive cruise control, namely the SafeDistance function. The safety chain for this SafeDistance safety function is shown in Figure 4. This safety chain is decomposed in four compo- nents, each on a different level of abstraction. More specifically, each component is unit tested up till a certain test integration level. This knowledge is for- mally captured in the corresponding artifact-view of Figure 6: Model-based functional safety test process each component. The artifact-view of the complete performed after MiL, (ii)Pil and Hil need to be se- SafeDistance safety function is shown graphically in quentially performed after SiL, (iii)Pil and HiL can be Figure 5. This defines that the AccMonitor compo- performed in parallel and (iiii) integration tests need nent, leftmost artifact-view in Figure 5 and highlighted to be sequential performed after the completion of the in the red box in Figure 4, is unit tested up till the soft- unit tests of all integrated components. ware integration level. To complete the unit test for this component, the tests on the processor and hard- With the artifact-views and the process rules ware integration level need to be performed. From this defined, a valid test strategy process model can graphical representation, the artifacts are not explic- automatically be generated. This text-to-model itly defined2 as they are implicit present in the activity transformation generates a directed graph, as shown names, namely: in Figure 6, where the test activities are depicted as nodes and the control flow as links. • Model-in-the-Loop: Input artifact is a model of the AccMonitor In this use case, we not only generate a directed • Software-in-the-Loop: Input artifact is source graph, we also generate a process model in a Causal code of the AccMonitor Block Diagram (CBD) formalism in Simulink c , which allows us to visually select the proper V&V test meth- • Processor/Hardware-in-the-Loop: Input ar- ods for the test activities, when necessary, to truly tifact is production code of the AccMonitor customize the test strategy process. An example of a test activity in CBD formalism is shown in Figure 7, Besides the necessary knowledge about the test where the selection of the V&V test methods, if appli- completion, we also define general process rules for cable, is present for the sub-activities. The available a valid execution flow of the model-based test activ- V&V test methods and the possible test case deriva- ities. We define that (i)SiL needs to be sequentially tion methods are compliant to the ISO/DIS 26262 as 2 Artifacts are also not explicitly present in the processes shown by the SpecifiedBy relation between the tables, Figure 7: Configurable functional safety test activity and ISO/DIS 26262 compliance relation originating from ISO/DIS 26262 - part 6, and the cus- tomizable CBD block. At this level, process rules are defined to capture the dependencies between the different test activities and their corresponding V&V test methods. The following list is a subset of the outcome of the applied process rules for this use case: Figure 8: Generated valid, optimized and customized test strategy • Resource usage test not feasible on MiL 5 Conclusion and future work • Back-to-back test not feasible on MiL This paper presents a methodology to facilitate the • Requirements based test on MiL forces require- model-based validation and verification of safety ments based test and boundary values test on SiL critical systems by (semi-)automatically generating valid, optimized test strategy processes compliant • PiL not feasible for unit testing to the ISO/DIS 26262 functional safety standard using an artifact-centric approach. We applied this • HiL not feasible for unit testing methodology to a well-known advanced driver assis- tance system, the adaptive cruise control, extended with the needed safety functions. This case study is After selecting the proper V&V test methods3 small-scale and yet complex enough to be suited for and applying these process rules, a valid, optimized future research. and customized test strategy model is automatically generated. In the future, we plan to extend the proposed methodology by introducing one or more design space An example of a generated test strategy model is exploration (DSE)algorithms to further optimize the shown in Figure 8. In this generated test strategy generation of valid test strategy processes. By taking process,the proper V&V test methods are selected the resource constraints, such as shared real-time and the above mentioned process rules are applied. To hardware platforms or human resources, into account, increase the readability and usability of the generated both a valid and optimized test strategy process test strategy model, we grouped the sub-activities of and an optimized test scheduling can be generated, each test activity at the particular integration level. which is beneficially for the overall verification and validation processes. Second, we want to extend the usage of the 3 The applied process rules have also influence on the avail- artifact-view by tightly coupling this artifact-view to able/valid V&V test methods the model information or meta-knowledge[Sirin14], enabling the use in design processes other than safety- [IEC61508] IEC 61508 ed. 2.0. International Elec- related automotive development. More specifically, trotechnical Commission, IEC;, 2009. we will include this information of the artifact-view within the validity frame[Denil17][Klikovits17] encap- [ISO/DIS26262] ISO/DIS 26262 (all parts), road ve- sulated with each component. This will increase the hicles functional safety. International Orga- (re-)usability of the components within the design nization for Standardization. processes. [Klikovits17] Klikovits, S., Denil, J., Muzy, A., & Salay, R. Modeling frames MoDeVVa Work- Lastly, we will compare our proposed methodology shop Proceedings, 2017. against more traditional activity-based processes to empirically assess the usability, correctness and scala- [Kuhrmann14] Kuhrmann, M., & Beecham, S. bility of our methodology. Artifact-Based Software Process Improve- ment and Management : A Method Pro- 6 Acknowledgments posal. 2018. This work has been carried out within the framework [Lee08] E. Lee et al., Cyber physical systems: De- of Flanders Make’s ICON project ’EnPower’ (TCO op- sign challenges, Object Oriented Real-Time timal system design for Energy and Power storage in Distributed Computing (ISORC), 2008 11th dynamic load applications, HBC.2016.0462) funded by IEEE International Symposium on. IEEE, the agency Flanders Innovation & Entrepreneurship pp. 363369, 2008. (VLAIO) and Flanders Make. Flanders Make is the [Nardi17] Nardi, A., & Armato, A. Functional safety Flemish strategic research centre for the manufactur- methodologies for automotive applications. ing industry. IEEE/ACM International Conference on Computer-Aided Design, Digest of Techni- References cal Papers, ICCAD, 2017Novem, 970975. https://doi.org/10.1109/ICCAD.2017.8203886, [Arc17] Arcaini, P., Gargantini, A., & Riccobene, 2017. E. Rigorous development process of a safety-critical system: from ASM models to [Sirin14] G. Sirin, C. J. Paredis, B. Yannou, E. Java code. International Journal on Soft- Coatanea, and E. Landel, A Model Identity ware Tools for Technology Transfer, 19(2), Card to Support Simulation Model Develop- 247269. https://doi.org/10.1007/s10009- ment Process in a Collaborative Multidisci- 015-0394-x, 2017. plinary Design Environment, 2014. [Baresi16] Baresi, L., Meroni, G., & Plebani, P. [Yongchareon18] Yongchareon, S., Jian, Y., & On Handling Business Process Anomalies Zhao, X. A View Framework for Mod- through Artifact-based Modeling, 2016. eling and Change Validation of Artifact- Centric Inter-Organizational Business [Bellotti10] Bellotti, M., & Mariani, R. How future Processes, Information Systems (April). automotive functional safety requirements https://doi.org/10.1016/j.is.2014.07.004, will impact microprocessors design. Micro- 2018. electronics Reliability, 50(911), 13201326. https://doi.org/10.1016/j.microrel.2010.07.041, [Zander11] Zander, J., Schieferdecker, I., & Moster- 2010. man, P. A Taxonomy of Model-Based Testing for Embedded Systems from [Conrad12] Conrad, M. Verification and Validation Multiple Industry Domains. Model- According to ISO 26262 : A Workflow to Based Testing for Embedded Systems, 122. Facilitate the Development of High-Integrity https://doi.org/doi:10.1201/b11321-2, 2011. Software. roc. ERTS 2012 Embedded Real Time Software Ans Systems., 2012. [Denil17] J. Denil, S. Klikovits, P. J. Mosterman, A. Vallecillo, and H. Vangheluwe, The exper- iment model and validity frame in M&S, Proceedings of the Symposium on Theory of Modelling and Simulation, 2017.