-Software for self-driving vehicles requires intensive importance to enable agile development. Meeting all these testing to avoid fatal accidents and to allow correct operation requirements represents the challenge that we want to tackle in real-world environments. Simulation frameworks allow to in this paper. ivmehitiacltees tuhseinbgehsaimvipoluifireodf mcoomdepllsexofsythsteemresalsuwcohrlads. aHuetnocneo,mthoueys A significant number of simulation frameworks already are important tools allowing to extend component and functional exists. In Section III, we compare the most popular and tests to address interconnections between sensors, actuators, powerful ones, such as Gazebo, PTV Vissim, SUMO, and Carand controllers in virtual and predefined environments. Existing Sim. Each of the investigated simulation frameworks exhibits simulators can be separated into high-level and low-level ones. its particular benefits and drawbacks. For example, Gazebo, fBoorthadadrreedsseisniggnaeldl dforrivvinergyssitpueactiifiocnss.ceWnahrilieoshaignhd-laerveelnsoitmsuuliatatobrles which is a highly versatile open-source simulation tool feaare suitable for mastering large testing environments such as turing multiple physics engines, does not support environment cities, they lack fine-grained simulation capabilities, e.g., turning data imports from OpenStreetMap. Conversely, PTV Vissim of wheels. In contrast, low-level simulators provide a high level of can import OpenStreetMap data but does not have a physics detail with realistic motion profiles. This is usually only possible engine and is relatively expensive. To our knowledge no ianppsrmoaacllh ttehsattincgomenbvinireosntmheenbtesn.eIfints tohfisboptahpherig,hw-leeveplreasnedntloawn- simulation framework supports native execution of Component level simulators to execute component and connector models. & Connector (C&C) models (Simulink, Modelica, LabView, Vehicle and traffic engineers can choose the most suitable level of EmbeddedMontiArc, and SysML are popular C&C representadetail for their application and integrate real-world environment tives) in realistic city environments. In fact, C&C modeling is data from OpenStreetMap. Moreover, the simulator allows for possible in MathWorks Simulink with Animation3D. However, iandcalputdaitnigonnseawndseenxstoernss,ioancstuoaftotrhse apnhdyscicoanltrvoelhiscylsetecmons.figAunroatthioenr no realistic city environment is provided by this solution. feature of our simulator is its automated testing support and its Finally, most simulators do not explicitly address automated ability to visualize 3D simulations in a browser. testing support. Index Terms-Simulation framework, self-driving vehicles, Even though further work is still needed, the simulation component and connector models, executable modeling framework introduced in this paper aims to combine all desirable features of the considered frameworks and to simultaneI. INTRODUCTION ously eliminate their drawbacks. We propose a MontiCAR [13] Autonomous vehicles are complex software and hardware simulation framework named MontiSim that meets the presystems providing a wide spectrum of safety-relevant functions viously highlighted requirements, thereby supporting model[1] which have to be tested and validated extensively before based software engineering (MBSE), test driven development they can be made available for series production. Real world (TDD), evolution, and execution of autonomous vehicle functests are not only dangerous but also time consuming and tions. Our framework has its own extensible physics engine cost-intensive and therefore unfeasible in early development and is compatible with OpenStreetMap enabling a realistic stages. This leads to the importance of simulation frameworks simulation of large-scale environments such as cities. At the for self-driving vehicles enabling safe and efficient testing in same time, highly-detailed low-level simulations of vehicle predefined environments. dynamics and accidents are possible. A further feature is the Ideally, a homogeneous and versatile solution would cover a continuous integration and regression testing support allowing large variety of scenarios starting from fine grained sensor and to perform unit, integration and acceptance testing. actuator simulation through to high level traffic scenarios. As This paper is structured as follows: First, we introduce a runa consequence such a versatile simulator would need to meet ning example and requirements on the simulator in Section II a broad set of requirements. First of all, it should combine the in order to demonstrate the capabilities of the developed framefeatures of high-level and low-level simulators. In other words, work. Second, we compare existing simulation frameworks it should be able to simulate large-scale environments but, at with our proposed solution in Section III. Third, in Section IV the same time, guarantee a high level of detail. Furthermore, we present our first contribution, namely, the modular archiit should allow the detailed simulation of real-world situations tecture of our simulation framework allowing to exchange such as city areas. Moreover, it needs to support an easy and adapt C&C controllers, sensors and actuators in integration and comparison of different sensors, actuators and vehicle models as well as maps and environment data controllers. In addition, automated regression testing is of great in simulation models. Our second contribution presented in
Section V are automatically executable testing concepts for C&C models such as regression tests with and without the environment. Finally, Section VI concludes this paper.
Based on previous research projects with industry partners
[
In the remainder of this section, the usage workflow of our proposed simulation framework is demonstrated based on a small realistic example, where two cars follow a straight street; videos showing our simulation framework in action are available at: https://youtu.be/TwMtA6ZV86I, https://youtu.be/ d-EEs62-rGM, and https://youtu.be/HiaHf0dd_1w.
(1) First, the map of interest is downloaded from
openstreetmap.org; based on this information the simulator
accurately recreates streets, buildings and traffic signs. (2)
Next, a simulation model as shown in Figure 1 is created.
It contains an environment description as well as
simulation parameters such as location, time, resolution, weather
conditions, and available vehicles. (3) New vehicle models
can be created as shown in Figure 2 if needed. Here, one
specific car is assembled by assigning its physical and visual
properties as well as its controlling unit, sensors and actuators.
For each input port of the controller, our framework delivers
three default virtual sensors with different noise models. (4)
The logic of the self-driving vehicle is modeled as a C&C
controller mapping the sensor inputs to actuator commands.
Therefore, we use the MontiCAR C&C modeling language
family together with its embedded math expression language
for behavior definitions [
Using appropriate quality measures such as the mean squared error (MSE) of the car’s trajectory, our simulation framework allows for an easy comparison between different C&C controllers.
Simulation frameworks represent a consistently used tool in
the software development and testing process of self-driving
vehicles. A wide variety of frameworks already exists on the
market. In this section, the most popular ones will be compared
in terms of their functionalities and capability to satisfy the
requirements presented in section II. In particular, we will
focus on Pelops [
(R1) Import and reuse of existing real world
environment data: Pelops [
(R2) Capability to simulate large-scale everyday
scenarios: This is a typical feature of high-level traffic simulators
which allows, for instance, to analyze different traffic densities
or different weather conditions. PTV Vissim [
(R3) Support for realistic and extensible car models
with sensors, controllers and actuators: Being able to
simulate different car, sensor, and controller models is a core
feature of our framework. Among the considered simulation
frameworks, only SUMO, PTV Vissim and SimTram do not
offer this feature due to their pure high-level nature. [
EMA
component LaneKeepingController {
ports in Q(0m:0.1m:5m) d[
out Q(-45°:0.2°:45°) s;
(R4) Multi-platform and portable devices support: CarSim, PTV Vissim, and Prescan exclusively support Windows machines. All other frameworks support Linux, as well. Mathworks Simulink 3D Animation supports also a 3D animation web viewer based on HTML5.
(R5) Automated support for continuous integration and regression testing: This requirement is only fulfilled by Gazebo and the OpenDaVinci framework. Gazebo runs a variety of regression tests concerning both the simulation and the physical robot either on an own cluster or in its Hudson regression test suite. OpenDaVinci itself is developed using continuous integration on Jenkins. Furthermore, its modular structure facilitates unit testing of new self-driving vehicle algorithms.
(R6) Physics engine: While Pelops, Prescan, Gazebo and CarSim provide their own physics engines, Mathworks Simulink 3D Animation does not include out-of-the-box physics. However, it allows to import a car modeled in Simulink and to simulate it. The Udacity simulator, RTMaps, SUMO, PTV Vissim, OpenDavinci, and SimTram do not provide a real physics engine.
(R7) 3D visualization for demonstration purposes: In the context of the considered frameworks, SUMO and SimTram are the only frameworks not supporting a 3D visualization.
Table I summarizes the comparison between all the presented frameworks including ours.
IV. FRAMEWORK FOR THE SIMULATION OF C&C MODELS
IN REALISTIC ENVIRONMENTS
An overview of the proposed MontiSim framework is shown in Figure 6. The main features provided include the support for rigid body based physics simulation, computer vision, versatile sensors, control systems, actuators, car models, real environment data as well as continuous integration and regression testing. The framework is based on the discrete time paradigm, i.e., the state of the simulation is updated in predefined discrete time steps. Furthermore, the main simulator can be coupled with an arbitrary amount of helper simulators in order to add specific capabilities such as vehicle to vehicle communication or tire pressure simulation. Helper simulators can be both discrete time or discrete event based.
The main part contains one or multiple car models describing a particular car and one or multiple MontiCar C&C models to model control systems. These models are independently developed by the vehicle engineer and the controller developer. A test case designer creates one or multiple simulation models describing the simulation. Furthermore, additional environment data can be inserted. All sensors and actuators are modeled and integrated by a domain expert. Apart from the MontiCar C&C model, each model is directly inserted into the simulation MontiCar Simulation Visualization raFm onM ew it
S ro im k
AcAAtcucttuuaaatttooorrr SAeAccnttuusaattooorrr
Car Assembler
Car Model framework. The MontiCar C&C model is translated to multithreaded C++ code, which is executed to control a vehicle during the simulation.
A big advantage of the presented simulation framework is the fact that all C&C languages can relatively easily be translated into another C&C language. Therefore, we will from now on refer to only one representative C&C language, MontiCar, which we developed.
The visualization part of the simulation framework consists of 3D models, which are created by a 3D designer and used in the visualization of the simulation. At the current state of development, it is possible to fluently simulate 2D scenarios with up to 50 cars and 3D scenarios with up to 5 cars.
In the following, each element of the simulation framework is explained in detail. 1 stream StearingAngleTest Stream 2 for Sensors2CarState Values for input port d (L=3m, w=2m) { 3 d = [50cm 50cm 2.5m 2.5m] 4 tick [50cm 1.3m 2.6m 1.8m]; Expected Values for output ports time step 5 phi = 0° tick 15.45°+/-0.05°; 6 y = - tick -; 15.4° ≤ exp. 7 }Do not care about value value ≤ 15.5°
Fig. 7: Stream Test Model.
TestingSensors
One main requirement for the MontiCAR C++ Code generator is to be compatible with the widely used Simulink. If we transform Simulink block diagrams into MontiCAR C&C models, the MontiCAR model will produce for the same input 7Rvalues the same output values as the original Simulink model. ) n This allows us to integrate Simulink models in our toolchain. ( o tiza In the first step, our C++ code generator calculates the exela cution order of the C&C component instances. This creates a i u iVsorted execution list, equivalent to Simulink slist command, s Dwhere only atomic components are considered. All other C&C 3 p components are similar to the Simulink virtual subsystems p and are flattened so that only their atomic components are p considered. The calculation of the execution order, which p is mostly not unique, satisfies the two main rules: (i) If p component C1 is connected with component C2, i.e, there p is at least one connector from any outgoing port of C1 to - any ingoing port of C2, than the execution order of C2 must pp be greater than the execution order of C1. (ii) Components p having no input ports (e.g. constant components) can have p any execution order as long as they satisfy (i). p The C++ code generator now uses the execution order to p invoke the calculation methods of the atomic blocks in the right order. In this context, the input values are pointers of the previously executed calculation method results so that objects do not need to be copied.
In MontiCAR, the behavior of atomic blocks is described
via MontiMath, a strongly typed Matlab-like Domain Specific
Language (DSL). To enable high-speed matrix operations, the
body of the calculation methods invokes the highly optimized
Octave [
To tackle the issue that one atomic component A can produce an output x in km/h and the other one B wants input values in m/s, the C++ code generator converts all input and output values to default SI unit values. In this example x is divided by 3:6 at end of the calculation method before being transferred to B. Thereby, the modeler benefits from SI unit support including generator errors, e.g., when trying to connect kg with m/s, as well as optimized Basic Linear Algebra Subprograms (BLAS) libraries.
The simulation of the environment relies upon OpenStreetMap which is a 2D model. Thus the third dimension has to be obtained from a different source or needs to be sampled as a random variable from an appropriate probability distribution.
In the generated map, for each intersection road signs and traffic lights are randomly generated, as well. Similarly, pedestrians can be included. Their behavior is defined by a set of five parameters which define their movement between two defined points P0 and P1: (1) the distance the pedestrian moves during each time frame (2) the side of the road on which the pedestrian is (3) the direction, either from P0 to P1 or the opposite (4) whether the pedestrian will cross the road or not (5) the instantaneous position of the pedestrian. The weather conditions can be either fixed, constantly or randomly changing.
Vehicles have access to the simulated environment via sensors implementing the Sensor interface; available sensor classes include, e.g., a WeatherSensor, LocationSensor, CameraSensor, CompassSensor, SpeedSensor, SteeringAngleSensor, DistanceToRightSensor. Adding or modifying sensors can be done easily.
C. Aggregation of C++ Code and Environment in Simulator / Physics Engine
The actuator, as shown in Figure 3, can either be written in Java or in MontiMath. Since the Actuator interface for MontiMath has one input and one output port of the same type, the simulator engine generates one large C&C model, which has the controller and all MontiMath actuators as child components. According to the car assembly model, the controller output ports are connected with the actuator inputs. All input ports and unused output ports of the controller are directly connected to the large C&C model, also all output ports of the actuators are connected to the corresponding output ports of the large C&C model. Due to the Actuator interface, the large C&C model has always the same interface as the controller. The large C&C model is now translated to C++ code in order to take advantage of the BLAS operations. Note that the simulator, the sensors and actuators are written in Java.
The generic Java Sensor interface contains one method that has two pointers as parameters, one to the complete world object and one to the current car object. This method returns one calculated value. Based on the external simulated environment and objects, the sensor can calculate an output. The generic Java Actuator interface contains also one method with a HashMap containing all input and output values of the C&C controller, e.g., the engine actuator for the acceleration does not only depend on the target acceleration but also on the current speed. In contrast to the generated C++ code, where all units are compatible by construction and thus ignored, the Java code for sensors and actuators works with the JScience library. In this way, unit incompatible computations result in Java compile errors.
The simulator (1) executes the sensors models, than (2) it transforms all JScience values to standard SI values and finally it (3) invokes the generated C++ code by JNI. Afterwards, the unitless C++ code result values of the large C&C model are (4) translated back to JScience values. Subsequently, the (5) Java actuators are executed and, eventually, (6) the physics engine updates all objects (vehicles, pedestrians) in the world. When this process is completed, everything starts from (1) again.
For systems or acceptance tests, it is helpful to visualize the simulated data. This way, it is possible to figure out missing unit or integration tests, such as "the car should not to leave the road", "the car should not wobble too much", "the car should not park too close in parking slot, otherwise the door cannot open".
Another important reason that justifies the visualization is the test case designer’s motivation, i.e., they produce models of better quality when the executed result can be seen. A visualization allows to demonstrate their models’ features, to actually see the performance of their automated vehicles and to better compare different models in order to identify the best one.
The visualization is realized as follows. The server sends the states of the objects (cars and pedestrians) to the ThreeJS library via web sockets for rendering the world. In order to have a fluent visualization, the JavaScript visualization script interpolates the car states, so that it can fluently drive between two discrete state updates preventing the car from hopping from one place to another. Technically, the visualization consists of two parts, an init to create the world and a loop part to move the objects.
a) Init.: The simulator delivers all data required to build up the entire map at the beginning, which contains (i) the terrain as bounds and height-map data, (ii) the street as 4D array (x,y,z position and street width) of nodes, (iii) the street signs, delivered as a JSON file for the geometry, a PNG image file as well as a 3D position point in the map and a 3D vector telling in what direction the sign points, (iv) the traffic lights, handled similarly to the street signs. To make the scenes more realistic, the visualization supports a day and night cycle and three different light sources: ambient light (that globally illuminates all objects in the scene), directional light (that gets emitted in a specific direction), hemisphere light (positioned directly above the scene).
b) Loop.: In the loop, the visualization sets the position of the cars and the pedestrians. In order to have a fluent visualization at 60 fps, which is independent from the steps per second that can be calculated by the simulator, the visualization has its own simple interpolation algorithms for positions and directions of all moved objects. This way the simulation looks fluent even if a position must be retransmitted due to data loss.
Since the simulator only considers the friction coefficient of the road, it does not model rain drops to save computational power on the server. Thus, the JavaScript web client contains a rain controller that creates a specified amount of rain or snow particles (which may have independent sizes) randomly in the world and moves the existing particles around, e.g., by just decrementing the y coordinate.
V. C&C UNIT, INTEGRATION, AND SYSTEM TESTING Apart from the simulation, MontiSim addresses testing purposes in order to validate sensors, controllers, actuators, as well as other components. Hence, in the remainder of this section, two testing methods are presented. First, a unit testing method without environment is presented. Second, a C&C integration test method is explained to test controllers with the environment.
A. C&C Unit, and Integration Testing without Environment
The aim of C&C black-box unit testing, similar to JUnit testing, is to validate the correctness of single C&C components by testing the component with given time-dependent input values and comparing its calculated time-dependent output values with the specified ones given in the stream test file. This methodology provides an easy and intuitive way of testing C&C components. To perform, e.g., in Simulink, a similar unit testing for subsystems (analogue to C&C components) the tester needs to execute the following steps for each test case: (1) create a new Simulink model as a new slx file, (2) import the subsystem to be tested as model reference, (3) add two signal builder blocks and manually specify (or via Matlab script) the time-dependent input values and the expected output values, (4) manually connect all output signals of the first signal builder to the input values of the model reference block, (5) compare each output of the model reference with the expected values of the second signal builder block by connecting both to the input ports of an equal subsystem, (6) connect the output ports of all equal blocks with the Simulink stop block, so that the simulation and thus the test will fail if the calculated output and the expected output are not the same. Our experience in several industry projects as well as in student labs showed that, due to the high technical obstacles, nearly no unit testing of subsystems is done in Simulink; and even then no automatic regression testing on build servers is done. As a consequence, errors are only detected later by inspecting the output values manually in a time-intensive testing phase.
This paragraph now explains how the C&C steering controller (Sensors2CarState component (s. l. 2)) can be tested in our simulation framework. For this controller, Figure 7 shows a test case for testing the correct calculation of the steering angle (phi) of the Sensors2CarState component. In this example, the input values for which the component is tested are defined in ll.3-4. Note that the tick represents one time step, e.g., going from t = 0s to t = 0:1s. The expected output values for phi, i.e., the calculated angle, are defined in l.5. as 15.45 +/-0.05 meaning that any value between 15.4 and 15.5 is accepted. Finally, the expected output values for y, i.e., the relative y-position of the car are defined in l.6. Since these values are irrelevant for this test, both time steps are defined by dashes.
If a unit test is successful, the result is a list of components used in this test case. Based on this information, a test coverage is computed. If a unit test fails, then the list of failing components is presented. Note that the inner-defined C&C components are regarded, as well; testing a large C&C component with many inner subcomponents is mostly called black-box integration testing.
B. C&C Integration, and System Testing with Environment
Unit testing controllers, sensors, or actuators can also be done in an environment to simulate real world scenarios. For example, consider the example in Figure 8. Here the car should follow the straight line autonomously. However, the path computed by the controller is not ideal as shown in the figure.
To test different controllers, the deviation from the optimal path can be computed for each controller. Based on this deviation, the most optimal controller can be identified. In this testing scenario, the deviations are totalized and compared. Starting from an ideal position on the line, the environment is initialized (void init(World w)). After each simulation time step (void simulateTick(World w)), the deviation from the ideal position is added to the previous deviation. When the simulation is finished (void finish(World w)), the total deviation is computed as deviation[m] s = and it is checked whether the driven distance[m] simulation was successful (boolean success()).
System tests including the environment enable
model-inthe-loop testing of liveness, smoothness, and responsiveness
of discretized closed-loop controllers [
Simulating self-driving vehicles is essential but still a highly complex task. Hence, in this paper, we presented a simulation framework that aims to address extension and adaptation concerns, real-world environments as well as real-world sensors and actuators. In particular, it provides an integration of highlevel and low-level simulation, which allows to analyze largescale scenarios as well as to address low-level details, e.g., the steering angle of the wheels. Furthermore, the framework supports integration and unit testing with and without the environment, respectively. This can be used to test sensors, actuators, and controllers. Based on a small scale example, we have demonstrated the applicability of the framework. Acknowledgements This research was supported by a Grant from the GIF, the GermanIsraeli Foundation for Scientific Research and Development, and by the Grant SPP1835 from DFG, the German Research Foundation.