Advanced driver assistance systems and (semi-)autonomous mobility systems will arguably be the biggest disruption of our everyday life in the next couple of years. The development of such systems comes with legal and technical challenges: Product liability regulations impose high standards on manufacturers regarding the safe operation of advanced driver assistance systems (ADAS). In the Automotive domain, su cient safety has yet to be proven through extensive and expensive testing. As a consequence, car manufacturers try to move testing e ort from the road to simulation. It is not clear, however, how results obtained from simulations transfer to the road. In this paper, we present an approach for leveraging simulation results during road tests. Our approach utilizes runtime monitors that are generated from speci cations, test scenarios, and simulated components. These monitors can be used during road tests and during operation for identifying untested situations and for checking functional correctness of an ADAS.
Driver assistance systems and (semi-)autonomous mobility systems are gaining
more importance in mobility carriers, such as vehicles, aircraft or rail- transport
systems. Guided by the vision of \zero accidents" (cf. [
However, today's conventional engineering methods are not adequate for providing such guarantees: Common vehicle eld tests are too expensive and cannot proof the system's dependability. They require too many miles to be driven in order to show that a system is su ciently safe. One way of reducing the cost of quality assurance is the application of structured testing as well as transferring a signi cant portion of the testing e ort to simulations. It is not clear, however,
Test
Scenarios
Real Behavior
how results obtained from simulations transfer to the road. Therefore, methods have to be developed that enable transferring results from simulations to the real world.
The DADAS1 project [
In this paper, we focus on the generation of monitors from design time
artifacts. We present a novel method for generating runtime monitors from
specications, test scenarios, and simulation components. The monitors can be used
during development or operation to collect information about untested
situations and unsafe behavior of an ADAS. Our main contribution is a conceptual
component-based architecture for these monitors along with methods for
generating the individual monitoring components from design time artifacts.
Related Work. In the automotive domain, runtime monitoring is primarily
used for the correctness and reliability of controllers for physical components.
The eld of diagnostics mainly uses supervision, fault detection and fault
management techniques based on physical parameters and mathematical models of
1 DADAS is an acronym for Dependable Advanced Driver Assistance Systems
the system's physical behavior to detect and resolve deviations and failures in the
component's behavior (cf. [
In academia, a wide range of runtime monitoring approaches for the
evaluation system properties exist. They can be categorized in di erent elds, based on
the objectives and property formalization. In the eld of requirements
monitoring aims to observe a system's runtime behavior in order to detect its deviations
from its requirements speci cation. The assumptions about requirements are
formalized in special languages, e.g. FLEA (cf. [
In general, testing and test coverage is not considered within runtime
monitoring elds, but some published works address the combination of testing and
runtime monitoring. In [
Outline. The remainder of this paper is organized as follows. In Section 2, we describe the conceptual architecture and elements of our envisaged runtime monitoring solution. Section 3 provides a case study based on the simulation of an lane change autopilot. We summarize our approach and point out future work in Sec. 4. 2
We are interested in discovering if an ADAS is operated within conditions that were tested to be safe in simulations. Untested operation conditions can be used as a basis for deriving further test scenarios. We are interested in identifying these conditions and in creating test scenarios from them. Our approach to this challenge is a combination of simulation at design time and monitoring at runtime.
In this section we present a novel conceptual architecture for monitors and a method for generating these monitors. In particular, we establish an interface World Sensor
Preprocessing
Processing Function
Output Postprocessing
Actuator
World between design time simulation and runtime monitoring that can be used to generate monitors from tested driving scenarios and help to identify new driving scenarios in data recorded from monitors.
Monitoring a complex ADAS. The general architecture of an ADAS can be described by the Input-Processing-Output (IPO) pattern and is shown in Fig. 2. The sensors of the vehicle acquire data from the environment. The data is then preprocessed in order to generate a single consistent interpretation of the current environmental situation (e.g., the identi ed objects on surrounding lanes). The main function of the ADAS analyzes this situation and determines necessary measures (e.g., changing lanes or adjusting speed) and sends corresponding commands to the output components. The commands are post-processed into set values for actuators (e.g., brakes, gearbox, or engine).
We are mainly interested in monitoring the main function of the ADAS to ensure that it operates within tested limits, i.e., for inputs known to be safe from tests. In order to transfer results obtained in simulations to the road, we have to monitor the preprocessing as well: We have to ensure that sensor inputs are preprocessed equivalently by components in the simulation and at runtime. We thus use a combination of two monitors: one for the preprocessing components and one for the main function.
The purpose of our monitors is to gradually ensure that an ADAS is tested to be safe in all relevant conditions. In case of untested conditions, we report these conditions for further testing. In this work we do not consider the application of appropriate measures during operation, e.g., disabling the ADAS, which is a highly complex issue in its own right. Moreover, we do not address the problem of guaranteeing functional safety of the complete system. This would require monitoring of all components and also a strategy for reaching a safe state in case of failure.
Generating monitors from design time artifacts. One of the main challenges in the DADAS scenario is nding a strategy for exchanging information between simulations and operation of an ADAS. In order for the proposed approach to become feasible in practice, we need to generalize from the data that is passed over the interfaces of the ADAS' components. If all our monitors would work at the level of physical signals, the monitors would almost never be able to identify a situation as tested. At the same time, we have to be very careful when generalizing. Otherwise we may identify conditions as safe wrongfully.
Specification t n e m n o r i v n E iltauno PrSeipmruolcaetsiosinng m i S
Test Scenarios
Function System under
Test
System under Operation Preprocessing Function
A sensible level of abstraction can usually be found in the speci cation of an ADAS (e.g., . . . on the lane to the right of the ego vehicle. . . ). We can use this level of abstraction as a basis for passing information from design time testing to operation and vice versa for two reasons.
{ The main function of the ADAS is developed against this speci cation. It should thus implement the speci cation and we can assume that it is safe to abstract from concrete data values to the level of the speci cation. In order to not solely rely on this assumption, we can additionally generate an abstract function from the speci cation and monitor the conformance of the main function of the ADAS to the speci cation at runtime. { The speci cation will often be the source for generating test scenarios and for measuring test coverage (of the speci cation). If we record situations at this level of abstraction during operation, we can directly feed those situations back into testing.
We have established that we generate two di erent monitors: one for the preprocessing and one for the main function. We have also discussed how these monitors are generated from design time artifacts. Figure 3 shows an overview of how monitoring components are generated from design time artifacts, in particular speci cations, test scenarios, and simulation components. We describe the architecture of the resulting monitors in the remainder of the section.
Monitoring the preprocessing components. We use components from the simulation to validate the preprocessing of the ADAS. We check the equivalence of the two sets of preprocessing components in order to ensure that tested scenarios transfer from simulation to the road. Please note that this is not a general veri cation of the data fusion performed by these components.
Figure 4 shows the architecture of the monitor for the preprocessing of an ADAS. The bottom line of components are part of the original ADAS' processing chain. In this setup, the real sensor data of the ADAS is mapped to the format of the input expected by preprocessing components in the simulation environment. We expect the two sets of preprocessing components to be almost identical, i.e., we assume that the simulation includes realistic preprocessing code and (simulated) sensor data. The results of the preprocessing of the ADAS is then compared to the results of the preprocessing from the simulation environment. In case the results di er, we cannot assume to be in a tested situation. Monitoring the main function of the ADAS. Figure 5 shows the components we use for monitoring the main function of the ADAS. The bottom line of components are part of the original ADAS' processing chain. We use components that abstract the input and output of the main function to the level of the speci cation. These abstract inputs and outputs are then used as a basis for monitoring functional correctness and for logging encountered unveri ed situations.
The monitors verify correctness by comparing the main function of the ADAS to the abstract function that is derived from the speci cation. The function is safe if the (abstracted) output of the main function is covered by the output of the abstract function, i.e., if the concrete output is allowed by the speci cation.
During development, the situation monitor logs the abstracted inputs of the main function, e.g. the situations, for which we are interested in generating more or better test scenarios during development. During operation, the monitor is equipped with a database of situations that were tested to be safe in simulations. In case the monitor observes an untested situation, no guarantees about the ADAS' safety can be made.
We have evaluated this concept for an architecture in a small case study successfully. The results from the evaluation are presented in the next section. Function
Logging Abstraction
Preliminary Evaluation: The Lane Change Autopilot In order to evaluate our monitoring concept for the main functionality of ADAS (cf. Section 2), we implemented a basic simulation environment and a simpli ed lane change autopilot according to a given speci cation. From the speci cation we also derived the components for the abstractions and the abstract function (cf. Fig. 5). We then used these components to test the feedback loop envisioned by the DADAS project: First, we simulated test scenarios and recorded safe situations. For this initial evaluation, we did not use a prototype vehicle (This will be done in a later step). Instead, we simulated random driving scenarios in order to nd untested situations and functional errors. The remainder of this Section covers the setup and results of this preliminary evaluation. Speci cation. We wrote a simple speci cation for the autopilot as a basis for the case study and restricted ourselves to overtaking on multi-lane roads. An existing speci cation of the behavior of a dead spot warning system served as guidance and example. As shown in Fig. 6, the speci cation assumes three road lanes and de nes one zone to be observed on each lane.
For the two neighboring lanes left and right of the ego vehicle, these zones cover the space that would be needed for a safe change to these lanes. The zone in front of the ego vehicle is used for triggering lane changes in our lane change autopilot. Our speci cation just requires that the autopilot does not overtake vehicles in its lane on their right side (this is illegal on German highways): If an object is observed in the zone in front of the vehicle, the autopilot may not initiate a lane change to its right lane.
Implementation. We have implemented two versions of the lane change function. One version is compliant with our speci cation while the other one is not. Both versions take a list of objects around the ego vehicle as input and process the target lane for the ego vehicle as output. We also implemented an abstract function in order to check the functional correctness of both versions at runtime.
While the architecture of our implementation closely resembles the components present in Fig. 2, we have abstracted from some of the details inside the single components. Our sensors, e.g., work with world objects instead with (a) (b) (c) (d) mileages of light waves and we have not implemented the execution of the lane change itself. On the other hand, since we were primarily interested in evaluating the architectural pattern presented in Sec. 2, we have implemented all monitoring components shown in Fig. 5.
Figure 7 shows the visualization of our simulation environment. The main component overlays the current abstract situation with the objects around the ego vehicle. Zones are colored red when they are occupied by a vehicle. The target lanes of the concrete and the abstract function are shown as blue (yellow respectively) points in front of the ego vehicle on the three lanes. While the yellow points mark the possible target lanes of the abstract function, does the blue point symbolize the concrete target lane of the ADAS.
Experiments. We have conducted a series of experiments during which we rst trained both versions of the lane change autopilot with a given test set of test scenarios. In a second step, we used the trained situation monitors while simulating random driving scenarios.
Figure 6 shows an excerpt of our initial set of test scenarios at design time used to train the situation monitors. In all displayed test scenarios, a vehicle enters the zone required for the lane change from behind or the front. In addition, some test scenarios place a vehicle in front of the ego vehicle in order to initiate a lane change. Figure 7b displays the set of situations that were recorded during testing by the situation monitor. Situations are pairs of zones on lanes (occupied or unoccupied) and possible target lanes.
Results. In our experiments, the functional correctness was checked correctly at the level of the abstract function. We were also able to train the situation monitor and then discover new (abstract) situations at runtime. These new situations could then be used as the basis for the iterative improvement of the test coverage by additional test scenarios. With increasing number of iterations, the most common situations had been tested and the test coverage of the ADAS reached a (a) Visualization
(b) Tested Situations Database stable level, when the monitor only recognized exceptional situations as untested situations.
These initial results indicate the feasibility of our approach for the runtime monitoring of ADAS. We are currently working on a re ned version of this case study, using the prototype of an autopilot with automatic lane change. The re ned version will be implemented in ADTF2 and simulated in VTD3 - two common tools for the development of ADAS. In the re ned case study, we are also evaluating how one can generate monitors for the preprocessing components of an ADAS using components from the simulation. 4
The DADAS approach addresses safety and correctness of advanced driver assistance systems. It consists of two parts - the veri cation of the ADAS in simulations for a nite set of de ned test scenarios at design time and the monitoring of the ADAS at runtime. The runtime monitoring ensures that the system and its environment remain within the behavior and situations which have been veri ed in simulations at design time. We have presented an conceptional framework for the runtime monitoring based on (i) decomposing the monitoring problem and on (ii) using and simulation components and artifacts for de ning monitors: In the presented approach the monitoring task is split into monitoring of the actual main function and monitoring the preprocessing components. We have presented a conceptual architecture for the necessary monitors and have discussed how the necessary components can be derived from design time artifacts. We have performed an initial evaluation of our approach using a small case study.
We are currently performing a second, larger case study on an automated lane changing autopilot. In this bigger case study, we have derived abstractions 2 ADTF is an acronym for Automotive Data and Time-Triggered Framework 3 VTD is an acronym for Virtual Test Drive and the abstract function successfully in a structured fashion. As a next step, we will try to automate the generation of monitoring components by using modelbased methods and evaluate their performance. We are currently investigating how to use components of the development and simulation environment (ADTF and VTD) for the equivalence check of the preprocessing components. The main goal of this second case study is to deploy our framework in a car eventually and to test the transfer from simulation to operation in the eld.