The complexity of software is rapidly increasing in many domains. Therefore, simulations have become established as a testing tool in recent years. Especially the virtual validation of autonomous systems leads to increasingly complex simulation environments. Nevertheless, the scenarios and the simulation results are not linked to the requirements. To close this gap, we develop a lightweight approach that allows the user to extract functional information. Simulation results can then be presented in di erent levels of detail in the original requirements. This replaces di cult translations of requirements and allows permanent comparison at all test levels.
selected information is used to control dynamic objects of the simulation.
To handle the large amount of data generated in the simulations, log les will be transferred to a model. The test engineers gain a clear representation of results and a quick overview of the correctly working software parts. The analysis of log les has two main points. Firstly, log les make it possible to go to selected time steps in the simulation and show the speci c situation. It facilitates the identi cation of the causes of errors. Secondly, accuracy of software can be checked in every time step. This helps to display the results of the simulation directly in the given requirements speci cations. In addition, this approach allows a quantitative analysis. Within the context of autonomous driving, it is possible to see values like driven kilometers, frequencies of activated functions, weather conditions, or manual corrections by the driver.
This new approach connects di erent types of requirements with a computer-based long-term simulation. This test method enables the execution of complex and realistic scenarios in which mandatory requirements in uence dynamic objects. Large-scale evaluation of log les is an innovative approach to test complex software. The results of this project can be useful for diverse research partners in several industries. This method analyzes whether the information in requirements documents re ects the complex situations in reality, and emphasizes shortcomings. It is a new approach to connect requirements and long-term simulations by means of lightweight extraction of information. The large-scale evaluation investigates various kinds of result presentations and possible conclusions for simulation execution. 2
The use of simulative test environments requires not only the modeling of the system under test but also a precise modeling of the scenario. Two essential approaches are in the focus of science. On the one hand, systematic approaches are explained in the works [BOS16, MKK15] in order to create scenarios step by step. On the other hand, the works of [BRSM16, IK14] are based on scenarios with especially critical situations. In both cases, requirements that derive from the functionality of the system are missing. To support scenario generation or automated test case generation, requirements are translated into structured models. Several authors propose di erent sets of sentence patterns that should be used to formulate requirements [EVFM16, MWHN09]. For natural language requirements, however, methods are missing which reliably support the scenario generation process. Alignment of requirements and test cases is a well-established eld of research and several solutions exist. Barmi et al. [BEF11] found that most studies of the subject were on model-based testing including a variety of formal methods for describing requirements with models or languages. One problem in this area is that the generated tests from the model cannot be executed directly against an implementation under test because they are on di erent levels of abstraction. In contrast to state of the art procedures, in my approach is no necessity for translation into executable languages [BLC+] or formulate requirements [MWHN09, EVFM16]. Therefore, the entire scope of simulation options of the natural language requirements remains without any translation loss. Another improvement of today's standards lies in the testability of software at any state of development. 3
My principal idea is schematically displayed in Figure 1. The starting point is a document of requirements formulated in natural language containing software speci cations. The present requirements are written without using pattern or other grammatical restrictions. Important keywords or sentence phrases are marked by the engineer and then extracted. These are matched with signals of the simulation and system, which is called a mapping. The creation of the simulation scenarios can be strongly supported by the information from the requirements. The engineers thus receive an overview of the necessary simulation properties (e.g. weather conditions, road characteristics or information about other road users) that must be included in the scenarios. In this way, the formulated requirements enter into the process of scenario generation. Depending on the simulators used and their connection, a partially automated or fully automated scenario generation is conceivable. Simulation results are output as log les, which in connection with the mapping results, are fed back to the original requirements document. In addition to log data from tra c simulations, it is also possible to use real driver log data. In this way, real log data can be matched with natural language requirements. The simulation results or real data are displayed directly in the originally analyzed phrases.
First behaviors of the software can be simulated with simple markings early in the development process. Especially new assistance systems or functions such as autonomous driving are very complex and can only be tested with complex simulations. The test engineers therefore need a lightweight approach to evaluate requirements without formal translation. This increases acceptance and does not restrict the requirement engineers.
R1: R2: R3:
A B C
SigA SigB SigC
There are two main motivations to use this approach: to nd information needed in order to choose a suitable simulation scenario and to check or monitor functionalities of a software component in di erent states of development.
In my work, I want to bring together natural language requirements and simulative test management. The methods used allow an evaluation of complex system simulations and a direct comparison to the legally binding requirement documents. 3.1
Research Questions The aim of this work is the review of natural language requirements by simulative long-term tests. In order to examine the various aspects of the approach, we have pre-set the following research questions: RQ1: Which scenario generation information can be derived from natural language requirements? RQ2: How many natural language requirements can be tested with simulations? RQ3: Can simulations ful ll the requirements equivalent to real test driving? RQ4: How can simulation results in natural language requirements support the testing process? To answer the research questions, I would like to develop an integrative test approach that links natural language requirements and simulations. The core idea is to let the engineer decide which software features are observed in the simulation. For this purpose, the requirements are not translated automatically but are intuitively marked by the engineer. 4
The research method has two main aspects. First, the natural language requirement must be easily accessible for the engineer to process. For this purpose, the requirements are initially marked manually using a markup language. To address RQ1, this tagged information is used to support scenario generation. The second main aspect of the approach is the processing of the log data. With this processing, RQ2 can be examined. Since the approach is independent of the simulation tools used, simulations at di erent test levels can be used. To answer RQ3, the log data from Hardware-in-the-Loop and Vehicle-in-the-Loop simulations are compared and displayed in the requirements. RQ4 will be answered as far as possible by qualitative surveys. The individual parts of the approach are explained in more detail below. 4.1
Markup Language For marking software functions and environment conditions, we developed a lightweight multilevel markup language to connect requirements speci cations and simulation runs. While there are many e orts within the research community to explicitly formalize requirements to improve on their validation possibilities [BEF11], this markup language aims to provide the requirements engineer with a means to intuitively annotate natural L1: Scope-Level
System
Environment l ita L2: Type-Level e D f o l e Le L3: Condition-Level v
L4: Causality-Level
Value{L1}
State{L1}
Event{L1}
Time Value{L1}-Condition State{L1} -Condition Event{L1} -Condition
Time -Condition {L3}-Trigger {L3}-Pre-Condition
{L3}-Action language requirements in order to unfold the implicitly contained information in a way it can be used for validation purposes within a simulation. The resulting language consists of elements, which are assigned to phrases in the natural language requirements documents with de ned content characteristics. This part of the process is performed by an engineer and is the starting point for the automated evaluation by the tool. Figure 2 shows, each element is assigned to one of four levels, which de ne the level of detail of the evaluation.
The Scope-Level is used to di erentiate between information on the system and on the simulation environment. As a result, the appearance of the objects in the simulation is displayed. However, no further information is available.
The Type-Level distinguishes the phrase of Level 1 into di erent types of text phrases depending on the behavior in the system. The di erent Level 2-types in uence the type of evaluation and are the basis for the de nition of conditions in Level 3.
The Condition-Level connects a type of Level 2 with a speci c value via comparison operators to create condition statements. However, the formulated conditions have no connection among each other.
The Causality-Level establishes a relationship between the conditions of Level 3 and creates causal relationships. This requires detailed knowledge of the system and the necessary work process performed by the user is time consuming. The result however is an in-depth evaluation. 4.2
Log Data as States The evaluation of the test systems, for example the vehicles, is realized by processing the log data. This has the decisive advantage that simulation steps can be summarized and understood as a status of the system. Every status can then be compared with the natural language requirements. Another advantage of the systematic evaluation of the log data is the traceability of the simulation process. The user can jump back to the simulated environment in case of occurring errors, special situations or vehicle-related maneuvers. In this way, the overall context can be examined more closely. The evaluation of the log data is independent of the test level used. If the simulations are used in di erent test stages, they can also be compared with each other. In relation to the V-model [FM92], which describes the development process of software in the automotive industry, the test levels Model-in-the-Loop, Software-in-the-Loop and Hardware-in-the-Loop can be compared with each other. Additionally, the log data of real trips can be compared with the simulation runs. 5 5.1
Marking Requirements For the extraction of simulation-relevant information, the engineer marks the important sentence phrases in the original requirements. The engineer can choose himself whether the information serves as a requirement for scenarios or the sentence phrases are tested in the simulation. In order not to have to mark all parts of the requirements so that evaluations are possible, a lightweight approach is used. It allows you to observe complete requirements or parts of them in the simulation at di erent levels of detail. Here we could show how any phrase phrases were marked by the user and evaluated in di erent levels of detail. To use this method, a tool is developed in which the natural language requirements are inserted. The tool allows the user to mark the requirements at the four previously presented levels. 5.2
Simulation Execution Software systems are becoming more complex and can no longer be validated by conventional single tests. Simulations are becoming an increasingly important tool for the validation of software systems for test management. Using long-term simulations and complex scenarios, a wide range of situations can be presented. VSimRTI [Sch11] links di erent simulators and provides a framework to simulate software from vehicles. As tra c simulator SUMO [BBEK11] is used, which works with real OpenStreetMaps data and tra c with up to thousands of vehicles. The modular design allows the extension, for example, by network simulators or selfdeveloped vehicle functions. This allows complex and realistic scenarios to be generated. When creating the scenarios, today the developers' experience ows in. Through the extracted information from the requirements, simulation properties can be systematically incorporated into the scenario generation and support an automated process. Other simulators have an increased focus on autonomous driving (Carla [DRC+17]) or simulating assistance functions and associated sensors (VTD [vNCNL+09]). Depending on the target system, the appropriate simulator can be used and is compatible with the presented approach. Regardless of the simulator used, natural language requirements can support scenario generation. The approach also allows an evaluation at the vehicle level or for all simulated vehicles in the scenario. 5.3
Representation of Evaluation An essential feedback mechanism is the presentation of the results in the original requirements document, depending on the chosen elements in the requirements and the analysis of the log data. Figure 3 shows the presentation of the evaluation results based on the simulation run in the presented example.
The evaluation shows that the granularity of the results also depends on the marking. In CL-1 the individual sentence phrases are evaluated, in CL-2 the results of the marked conditions are displayed and CL-3 shows a completely evaluated causal relationship.
The given example illustrates the in uence of the speci cation degree on possible evaluation options. For basic analysis or early system development stages, lower and less time-consuming evaluation levels are suitable. However the tool also includes more complex options of evaluation. Though increasing complexity requires an increasing e ort, evaluation and validation of entire requirements is possible. 5.4
Qualitative surveys With two planned qualitative surveys di erent aspects of the approach are to be evolved. In the rst part, the 4-level markup language is examined in more detail. The following questions are interesting: How many requirements can be expressed with the language? How many tags are possible within a request? How many requirements can not be covered by the simulation?
The second part focuses on the presentation of the results. Here, test engineers evaluate which values should be displayed in the speci cations and how a good representation of the results is possible. The results of the surveys have a direct in uence on the development of the approach. 6
According to a literature search in the area of simulative test validation and information extraction from requirement documents, gaps in the processing of natural language requirements could be identi ed. Subsequently, based on existing formalization requirements, a novel markup languange was developed. For easy user application, a tool has been developed which reads in natural language requirements, makes the developed language usable and makes markings possible. We presented a rst prototype at this year's REFSQ conference.
At this point in time, the marking process and the mapping are manual steps. Neural networks show promising results in previous work in requirements engineering. The automated detection of states, values and events with neural networks will also be investigated in the future. The engineer then gets a pre-marked speci cation and only needs to con rm the suggested markings.
In the future, the mapping should be integrated into the tool. An automated search for signal names that should support the engineer is planned. Building on this, the mapping can be fully automated using methods of machine learning. Today, the evaluation process of log data is already fully automated, but only limited to the evaluation of a simulation run. Based on this, the comparison of several log data sets should be the focus of research. Qualitative surveys aim to identify possible ways of presenting the results
The creation of scenarios of a long-term simulation is supported with presented approach and partially automated. The selected requirements have a direct in uence on the scenario generation. It is planned to compare two test methods for identifying errors and evaluating system parameters. For this purpose, a standard test process of a system function from the automotive industry is compared with a tra c simulation. In both cases the same controller will be used to compare the test methods in terms of their duration, cost and identi ed errors. An accompanying survey is planned to evaluate the context information (frequency of function triggering, simulation parameters, etc.).
Subsequently, the approach, in addition to the automotive sector, will be evaluated in a second domain. A second large area of application for simulations is software development in the smart factory area. Here the complexity of the scenario is very limited but due to the increasing networking of the systems, the focus is more on the simulation of message transmission. [BBEK11]
Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. Sumo{simulation of urban mobility. In The Third International Conference on Advances in System Simulation (SIMUL 2011), Barcelona, Spain, volume 42, 2011.
Z. A. Barmi, A. H. Ebrahimi, and R. Feldt. Alignment of requirements speci cation and testing: A systematic mapping study. In IEEE International Conference on Software Testing, Veri cation and Validation Workshops, 2011.
Barrett R. Bryant, Beurn-Seuk Lee, Fei Cao, Wei Zhao, and Je rey G. Gray. From natural language requirements to executable models of software components.
J. Bach, S. Otten, and E. Sax. Model based scenario speci cation for development and test of
automated drivin
J. Eckhardt, A. Vogelsang, H. Femmer, and P. Mager. Challenging incompleteness of performance
requirements by sentence patterns. In IEEE International Requirements En