=Paper=
{{Paper
|id=Vol-2376/DS-paper2
|storemode=property
|title=Requirements-based Simulation Execution for Virtual Validation of Autonomous Systems
|pdfUrl=https://ceur-ws.org/Vol-2376/DS-paper2.pdf
|volume=Vol-2376
|authors=Florian Pudlitz
|dblpUrl=https://dblp.org/rec/conf/refsq/Pudlitz19
}}
==Requirements-based Simulation Execution for Virtual Validation of Autonomous Systems==
https://ceur-ws.org/Vol-2376/DS-paper2.pdf
Requirements-based Simulation Execution for Virtual
Validation of Autonomous Systems
Florian Pudlitz
Technische Universität
Berlin, Germany
florian.pudlitz@tu-berlin.de
Abstract
The complexity of software is rapidly increasing in many domains.
Therefore, simulations have become established as a testing tool in
recent years. Especially the virtual validation of autonomous systems
leads to increasingly complex simulation environments. Nevertheless,
the scenarios and the simulation results are not linked to the require-
ments. To close this gap, we develop a lightweight approach that allows
the user to extract functional information. Simulation results can then
be presented in different levels of detail in the original requirements.
This replaces difficult translations of requirements and allows perma-
nent comparison at all test levels.
1 Introduction
In recent years, the number of complex and autonomous systems has increased significantly [PBKS07]. Most of
present innovations are driven by software solutions. Software components enable newest assistance systems of
autonomous driving as well as intelligent robots in medicine, or increased productivity in the context of Smart
Factory. Validating and testing the behavior of these complex systems is challenging because of their dynamic
nature and open context. New testing methods like simulations enable a fast, comprehensive, and automated test
implementation. Simulations can accurately model complex environments and the behavior of systems. Through
self-learning behavior models and dynamic adaptation-based systems, the scenarios in the simulations are less and
less defined at runtime. But, today’s computer-based simulations are standardized processes, focusing on small
specific situations. The used scenarios do not necessarily mirror reality to the full extent. The results of previous
simulations were limited to the detection of occurring software and signal errors. In limited cases, due to the
absence of errors, the correctness of the functionality can be inferred. Complete verification of requirements and
analysis of values in every time step are not performed. A further aspect is the selection of scenarios or simulation
objects. Currently, test engineers create simulations manually, but mostly those do not cover the complexity
of the entire Software. In my dissertation, I aim to develop a test environment for autonomous systems that
is based on simulations. It automatically connects requirements specifications with results of simulation runs.
The goal is an automated computer-based long-term simulation. This ensures that the simulation contains all
objects of the software description. Requirements specifications of software are often given in natural language
or models. A lightweight approach allows a fast and simple extraction of information for test engineers. The
Copyright c by the paper’s authors. Copying permitted for private and academic purposes.
In: P. Mäder, P. Spoletini (eds.): Joint Proceedings of REFSQ-2019 Workshops, Doctoral Symposium, Live Studies Track, and
Poster Track, Essen, Germany, 18-03-2019, published at http://ceur-ws.org
selected information is used to control dynamic objects of the simulation.
To handle the large amount of data generated in the simulations, log files will be transferred to a model. The
test engineers gain a clear representation of results and a quick overview of the correctly working software parts.
The analysis of log files has two main points. Firstly, log files make it possible to go to selected time steps in
the simulation and show the specific situation. It facilitates the identification of the causes of errors. Secondly,
accuracy of software can be checked in every time step. This helps to display the results of the simulation directly
in the given requirements specifications. In addition, this approach allows a quantitative analysis. Within the
context of autonomous driving, it is possible to see values like driven kilometers, frequencies of activated functions,
weather conditions, or manual corrections by the driver.
This new approach connects different types of requirements with a computer-based long-term simulation. This
test method enables the execution of complex and realistic scenarios in which mandatory requirements influence
dynamic objects. Large-scale evaluation of log files is an innovative approach to test complex software. The
results of this project can be useful for diverse research partners in several industries. This method analyzes
whether the information in requirements documents reflects the complex situations in reality, and emphasizes
shortcomings. It is a new approach to connect requirements and long-term simulations by means of lightweight
extraction of information. The large-scale evaluation investigates various kinds of result presentations and
possible conclusions for simulation execution.
2 Related Work
The use of simulative test environments requires not only the modeling of the system under test but also a precise
modeling of the scenario. Two essential approaches are in the focus of science. On the one hand, systematic
approaches are explained in the works [BOS16, MKK15] in order to create scenarios step by step. On the other
hand, the works of [BRSM16, IK14] are based on scenarios with especially critical situations. In both cases,
requirements that derive from the functionality of the system are missing. To support scenario generation or
automated test case generation, requirements are translated into structured models. Several authors propose
different sets of sentence patterns that should be used to formulate requirements [EVFM16, MWHN09]. For
natural language requirements, however, methods are missing which reliably support the scenario generation
process. Alignment of requirements and test cases is a well-established field of research and several solutions
exist. Barmi et al. [BEF11] found that most studies of the subject were on model-based testing including a variety
of formal methods for describing requirements with models or languages. One problem in this area is that the
generated tests from the model cannot be executed directly against an implementation under test because they
are on different levels of abstraction. In contrast to state of the art procedures, in my approach is no necessity
for translation into executable languages [BLC+ ] or formulate requirements [MWHN09, EVFM16]. Therefore,
the entire scope of simulation options of the natural language requirements remains without any translation loss.
Another improvement of today’s standards lies in the testability of software at any state of development.
3 Idea
My principal idea is schematically displayed in Figure 1. The starting point is a document of requirements
formulated in natural language containing software specifications. The present requirements are written without
using pattern or other grammatical restrictions. Important keywords or sentence phrases are marked by the
engineer and then extracted. These are matched with signals of the simulation and system, which is called
a mapping. The creation of the simulation scenarios can be strongly supported by the information from the
requirements. The engineers thus receive an overview of the necessary simulation properties (e.g. weather
conditions, road characteristics or information about other road users) that must be included in the scenarios.
In this way, the formulated requirements enter into the process of scenario generation. Depending on the
simulators used and their connection, a partially automated or fully automated scenario generation is conceivable.
Simulation results are output as log files, which in connection with the mapping results, are fed back to the original
requirements document. In addition to log data from traffic simulations, it is also possible to use real driver log
data. In this way, real log data can be matched with natural language requirements. The simulation results or
real data are displayed directly in the originally analyzed phrases.
First behaviors of the software can be simulated with simple markings early in the development process. Es-
pecially new assistance systems or functions such as autonomous driving are very complex and can only be tested
with complex simulations. The test engineers therefore need a lightweight approach to evaluate requirements
without formal translation. This increases acceptance and does not restrict the requirement engineers.
Requirements Mapping
Simulation
R1:
A SigA
B SigB
R2:
C SigC
R3:
Figure 1: Schematic representation of a requirements specification linked to a simulation with influencing inter-
mediate steps
There are two main motivations to use this approach: to find information needed in order to choose a
suitable simulation scenario and to check or monitor functionalities of a software component in different states
of development.
In my work, I want to bring together natural language requirements and simulative test management. The
methods used allow an evaluation of complex system simulations and a direct comparison to the legally binding
requirement documents.
3.1 Research Questions
The aim of this work is the review of natural language requirements by simulative long-term tests. In order to
examine the various aspects of the approach, we have pre-set the following research questions:
RQ1: Which scenario generation information can be derived from natural language requirements?
RQ2: How many natural language requirements can be tested with simulations?
RQ3: Can simulations fulfill the requirements equivalent to real test driving?
RQ4: How can simulation results in natural language requirements support the testing process?
To answer the research questions, I would like to develop an integrative test approach that links natural language
requirements and simulations. The core idea is to let the engineer decide which software features are observed in
the simulation. For this purpose, the requirements are not translated automatically but are intuitively marked
by the engineer.
4 Research Method
The research method has two main aspects. First, the natural language requirement must be easily accessible
for the engineer to process. For this purpose, the requirements are initially marked manually using a markup
language. To address RQ1, this tagged information is used to support scenario generation. The second main
aspect of the approach is the processing of the log data. With this processing, RQ2 can be examined. Since the
approach is independent of the simulation tools used, simulations at different test levels can be used. To answer
RQ3, the log data from Hardware-in-the-Loop and Vehicle-in-the-Loop simulations are compared and displayed
in the requirements. RQ4 will be answered as far as possible by qualitative surveys. The individual parts of the
approach are explained in more detail below.
4.1 Markup Language
For marking software functions and environment conditions, we developed a lightweight multilevel markup lan-
guage to connect requirements specifications and simulation runs. While there are many efforts within the
research community to explicitly formalize requirements to improve on their validation possibilities [BEF11],
this markup language aims to provide the requirements engineer with a means to intuitively annotate natural
L1: Scope-Level System Environment
Level of Detail
L2: Type-Level Value{L1} State{L1} Event{L1} Time
Value{L1}-Condition
L3: Condition-Level State{L1} -Condition
Event{L1} -Condition
Time -Condition
{L3}-Trigger
L4: Causality-Level {L3}-Pre-Condition
{L3}-Action
Figure 2: Overview of levels and elements
language requirements in order to unfold the implicitly contained information in a way it can be used for vali-
dation purposes within a simulation. The resulting language consists of elements, which are assigned to phrases
in the natural language requirements documents with defined content characteristics. This part of the process
is performed by an engineer and is the starting point for the automated evaluation by the tool. Figure 2 shows,
each element is assigned to one of four levels, which define the level of detail of the evaluation.
The Scope-Level is used to differentiate between information on the system and on the simulation environment.
As a result, the appearance of the objects in the simulation is displayed. However, no further information is
available.
The Type-Level distinguishes the phrase of Level 1 into different types of text phrases depending on the
behavior in the system. The different Level 2-types influence the type of evaluation and are the basis for the
definition of conditions in Level 3.
The Condition-Level connects a type of Level 2 with a specific value via comparison operators to create
condition statements. However, the formulated conditions have no connection among each other.
The Causality-Level establishes a relationship between the conditions of Level 3 and creates causal relation-
ships. This requires detailed knowledge of the system and the necessary work process performed by the user is
time consuming. The result however is an in-depth evaluation.
4.2 Log Data as States
The evaluation of the test systems, for example the vehicles, is realized by processing the log data. This has the
decisive advantage that simulation steps can be summarized and understood as a status of the system. Every
status can then be compared with the natural language requirements. Another advantage of the systematic
evaluation of the log data is the traceability of the simulation process. The user can jump back to the simulated
environment in case of occurring errors, special situations or vehicle-related maneuvers. In this way, the overall
context can be examined more closely. The evaluation of the log data is independent of the test level used.
If the simulations are used in different test stages, they can also be compared with each other. In relation to
the V-model [FM92], which describes the development process of software in the automotive industry, the test
levels Model-in-the-Loop, Software-in-the-Loop and Hardware-in-the-Loop can be compared with each other.
Additionally, the log data of real trips can be compared with the simulation runs.
5 Approach
5.1 Marking Requirements
For the extraction of simulation-relevant information, the engineer marks the important sentence phrases in the
original requirements. The engineer can choose himself whether the information serves as a requirement for
scenarios or the sentence phrases are tested in the simulation. In order not to have to mark all parts of the
requirements so that evaluations are possible, a lightweight approach is used. It allows you to observe complete
requirements or parts of them in the simulation at different levels of detail. Here we could show how any phrase
phrases were marked by the user and evaluated in different levels of detail. To use this method, a tool is developed
Figure 3: Resulting marks of a simulation evaluation
in which the natural language requirements are inserted. The tool allows the user to mark the requirements at
the four previously presented levels.
5.2 Simulation Execution
Software systems are becoming more complex and can no longer be validated by conventional single tests.
Simulations are becoming an increasingly important tool for the validation of software systems for test man-
agement. Using long-term simulations and complex scenarios, a wide range of situations can be presented.
VSimRTI [Sch11] links different simulators and provides a framework to simulate software from vehicles. As
traffic simulator SUMO [BBEK11] is used, which works with real OpenStreetMaps data and traffic with up to
thousands of vehicles. The modular design allows the extension, for example, by network simulators or self-
developed vehicle functions. This allows complex and realistic scenarios to be generated. When creating the
scenarios, today the developers’ experience flows in. Through the extracted information from the requirements,
simulation properties can be systematically incorporated into the scenario generation and support an automated
process. Other simulators have an increased focus on autonomous driving (Carla [DRC+ 17]) or simulating assis-
tance functions and associated sensors (VTD [vNCNL+ 09]). Depending on the target system, the appropriate
simulator can be used and is compatible with the presented approach. Regardless of the simulator used, natural
language requirements can support scenario generation. The approach also allows an evaluation at the vehicle
level or for all simulated vehicles in the scenario.
5.3 Representation of Evaluation
An essential feedback mechanism is the presentation of the results in the original requirements document, depend-
ing on the chosen elements in the requirements and the analysis of the log data. Figure 3 shows the presentation
of the evaluation results based on the simulation run in the presented example.
The evaluation shows that the granularity of the results also depends on the marking. In CL-1 the individual
sentence phrases are evaluated, in CL-2 the results of the marked conditions are displayed and CL-3 shows a
completely evaluated causal relationship.
The given example illustrates the influence of the specification degree on possible evaluation options. For
basic analysis or early system development stages, lower and less time-consuming evaluation levels are suitable.
However the tool also includes more complex options of evaluation. Though increasing complexity requires an
increasing effort, evaluation and validation of entire requirements is possible.
5.4 Qualitative surveys
With two planned qualitative surveys different aspects of the approach are to be evolved. In the first part,
the 4-level markup language is examined in more detail. The following questions are interesting: How many
requirements can be expressed with the language? How many tags are possible within a request? How many
requirements can not be covered by the simulation?
The second part focuses on the presentation of the results. Here, test engineers evaluate which values should
be displayed in the specifications and how a good representation of the results is possible. The results of the
surveys have a direct influence on the development of the approach.
6 Progress
According to a literature search in the area of simulative test validation and information extraction from require-
ment documents, gaps in the processing of natural language requirements could be identified. Subsequently,
based on existing formalization requirements, a novel markup languange was developed. For easy user applica-
tion, a tool has been developed which reads in natural language requirements, makes the developed language
usable and makes markings possible. We presented a first prototype at this year’s REFSQ conference.
At this point in time, the marking process and the mapping are manual steps. Neural networks show promising
results in previous work in requirements engineering. The automated detection of states, values and events with
neural networks will also be investigated in the future. The engineer then gets a pre-marked specification and
only needs to confirm the suggested markings.
In the future, the mapping should be integrated into the tool. An automated search for signal names that
should support the engineer is planned. Building on this, the mapping can be fully automated using methods
of machine learning. Today, the evaluation process of log data is already fully automated, but only limited to
the evaluation of a simulation run. Based on this, the comparison of several log data sets should be the focus of
research. Qualitative surveys aim to identify possible ways of presenting the results
The creation of scenarios of a long-term simulation is supported with presented approach and partially auto-
mated. The selected requirements have a direct influence on the scenario generation. It is planned to compare
two test methods for identifying errors and evaluating system parameters. For this purpose, a standard test
process of a system function from the automotive industry is compared with a traffic simulation. In both cases
the same controller will be used to compare the test methods in terms of their duration, cost and identified er-
rors. An accompanying survey is planned to evaluate the context information (frequency of function triggering,
simulation parameters, etc.).
Subsequently, the approach, in addition to the automotive sector, will be evaluated in a second domain. A
second large area of application for simulations is software development in the smart factory area. Here the
complexity of the scenario is very limited but due to the increasing networking of the systems, the focus is more
on the simulation of message transmission.
References
[BBEK11] Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. Sumo–simulation of
urban mobility. In The Third International Conference on Advances in System Simulation (SIMUL
2011), Barcelona, Spain, volume 42, 2011.
[BEF11] Z. A. Barmi, A. H. Ebrahimi, and R. Feldt. Alignment of requirements specification and testing:
A systematic mapping study. In IEEE International Conference on Software Testing, Verification
and Validation Workshops, 2011.
[BLC+ ] Barrett R. Bryant, Beurn-Seuk Lee, Fei Cao, Wei Zhao, and Jeffrey G. Gray. From natural language
requirements to executable models of software components.
[BOS16] J. Bach, S. Otten, and E. Sax. Model based scenario specification for development and test of
automated driving functions. In 2016 IEEE Intelligent Vehicles Symposium (IV), pages 1149–
1155, 2016.
[BRSM16] G. Bagschik, A. Reschka, T. Stolte, and M. Maurer. Identification of potential hazardous events
for an unmanned protective vehicle. In 2016 IEEE Intelligent Vehicles Symposium (IV), pages
691–697, 2016.
[DRC+ 17] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla:
An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning,
pages 1–16, 2017.
[EVFM16] J. Eckhardt, A. Vogelsang, H. Femmer, and P. Mager. Challenging incompleteness of performance
requirements by sentence patterns. In IEEE International Requirements Engineering Conference
(RE), 2016.
[FM92] Kevin Forsberg and Harold Mooz. The relationship of systems engineering to the project cycle.
Engineering Management Journal, 4(3):36–43, 1992.
[IK14] Masao Ito and Koichi Kishida. An approach to manage the concept phase of iso 26262. J. Softw.
Evol. Process, 26(9):829–836, September 2014.
[MKK15] P. Minnerup, T. Kessler, and A. Knoll. Collecting simulation scenarios by analyzing physical test
drives. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pages
2915–2920, Sep. 2015.
[MWHN09] A. Mavin, P. Wilkinson, A. Harwood, and M. Novak. Easy approach to requirements syntax (ears).
In 2009 17th IEEE International Requirements Engineering Conference, pages 317–322, 2009.
[PBKS07] A. Pretschner, M. Broy, I. H. Kruger, and T. Stauner. Software engineering for automotive systems:
A roadmap. In Future of Software Engineering, 2007. FOSE ’07, pages 55–71, 2007.
[Sch11] Björn Schünemann. V2x simulation runtime infrastructure vsimrti: An assessment tool to design
smart traffic management systems. Computer Networks, 55(14):3189–3198, 2011.
[vNCNL+ 09] Kilian von Neumann-Cosel, Mirko Nentwig, Daniel Lehmann, Johannes Speth, and Alois Knoll.
Preadjustment of a vision-based lane tracker - using virtual test drive wihtin a hardware in the
loop simulator. In Proceedings of the Driving Simulation Conference Monaco, 2009.