=Paper=
{{Paper
|id=Vol-2019/exe_5
|storemode=property
|title=Simulation Framework for Executing Component and Connector Models of Self-Driving Vehicles
|pdfUrl=https://ceur-ws.org/Vol-2019/exe_5.pdf
|volume=Vol-2019
|authors=Filippo Grazioli,Evgeny Kusmenko,Alexander Roth,Bernhard Rumpe,Michael von Wenckstern
|dblpUrl=https://dblp.org/rec/conf/models/GrazioliK0RW17
}}
==Simulation Framework for Executing Component and Connector Models of Self-Driving Vehicles==
https://ceur-ws.org/Vol-2019/exe_5.pdf
Simulation Framework for Executing Component
and Connector Models of Self-Driving Vehicles
Filippo Grazioli, Evgeny Kusmenko, Alexander Roth, Bernhard Rumpe, Michael von Wenckstern
Software Engineering, RWTH Aachen University, Germany
Abstract—Software for self-driving vehicles requires intensive importance to enable agile development. Meeting all these
testing to avoid fatal accidents and to allow correct operation requirements represents the challenge that we want to tackle
in real-world environments. Simulation frameworks allow to in this paper.
imitate the behaviour of complex systems such as autonomous
vehicles using simplified models of the real world. Hence, they A significant number of simulation frameworks already
are important tools allowing to extend component and functional exists. In Section III, we compare the most popular and
tests to address interconnections between sensors, actuators, powerful ones, such as Gazebo, PTV Vissim, SUMO, and Car-
and controllers in virtual and predefined environments. Existing Sim. Each of the investigated simulation frameworks exhibits
simulators can be separated into high-level and low-level ones. its particular benefits and drawbacks. For example, Gazebo,
Both are designed for very specific scenarios and are not suitable
for addressing all driving situations. While high-level simulators which is a highly versatile open-source simulation tool fea-
are suitable for mastering large testing environments such as turing multiple physics engines, does not support environment
cities, they lack fine-grained simulation capabilities, e.g., turning data imports from OpenStreetMap. Conversely, PTV Vissim
of wheels. In contrast, low-level simulators provide a high level of can import OpenStreetMap data but does not have a physics
detail with realistic motion profiles. This is usually only possible engine and is relatively expensive. To our knowledge no
in small testing environments. In this paper, we present an
approach that combines the benefits of both high-level and low- simulation framework supports native execution of Component
level simulators to execute component and connector models. & Connector (C&C) models (Simulink, Modelica, LabView,
Vehicle and traffic engineers can choose the most suitable level of EmbeddedMontiArc, and SysML are popular C&C representa-
detail for their application and integrate real-world environment tives) in realistic city environments. In fact, C&C modeling is
data from OpenStreetMap. Moreover, the simulator allows for possible in MathWorks Simulink with Animation3D. However,
adaptations and extensions of the physical vehicle configuration
including new sensors, actuators and control systems. Another no realistic city environment is provided by this solution.
feature of our simulator is its automated testing support and its Finally, most simulators do not explicitly address automated
ability to visualize 3D simulations in a browser. testing support.
Index Terms—Simulation framework, self-driving vehicles, Even though further work is still needed, the simulation
component and connector models, executable modeling framework introduced in this paper aims to combine all desir-
able features of the considered frameworks and to simultane-
I. I NTRODUCTION ously eliminate their drawbacks. We propose a MontiCAR [13]
Autonomous vehicles are complex software and hardware simulation framework named MontiSim that meets the pre-
systems providing a wide spectrum of safety-relevant functions viously highlighted requirements, thereby supporting model-
[1] which have to be tested and validated extensively before based software engineering (MBSE), test driven development
they can be made available for series production. Real world (TDD), evolution, and execution of autonomous vehicle func-
tests are not only dangerous but also time consuming and tions. Our framework has its own extensible physics engine
cost-intensive and therefore unfeasible in early development and is compatible with OpenStreetMap enabling a realistic
stages. This leads to the importance of simulation frameworks simulation of large-scale environments such as cities. At the
for self-driving vehicles enabling safe and efficient testing in same time, highly-detailed low-level simulations of vehicle
predefined environments. dynamics and accidents are possible. A further feature is the
Ideally, a homogeneous and versatile solution would cover a continuous integration and regression testing support allowing
large variety of scenarios starting from fine grained sensor and to perform unit, integration and acceptance testing.
actuator simulation through to high level traffic scenarios. As This paper is structured as follows: First, we introduce a run-
a consequence such a versatile simulator would need to meet ning example and requirements on the simulator in Section II
a broad set of requirements. First of all, it should combine the in order to demonstrate the capabilities of the developed frame-
features of high-level and low-level simulators. In other words, work. Second, we compare existing simulation frameworks
it should be able to simulate large-scale environments but, at with our proposed solution in Section III. Third, in Section IV
the same time, guarantee a high level of detail. Furthermore, we present our first contribution, namely, the modular archi-
it should allow the detailed simulation of real-world situations tecture of our simulation framework allowing to exchange
such as city areas. Moreover, it needs to support an easy and adapt C&C controllers, sensors and actuators in
integration and comparison of different sensors, actuators and vehicle models as well as maps and environment data
controllers. In addition, automated regression testing is of great in simulation models. Our second contribution presented in
Section V are automatically executable testing concepts for steering actuator modeled in MontiCAR. In l.12-13 zero-mean
C&C models such as regression tests with and without the normally distributed noise with a variance of 0.01 is added to
environment. Finally, Section VI concludes this paper. the manipulated variable of the actuator. Thereby, we take into
account its non-ideal and non-deterministic nature as well as
II. RUNNING E XAMPLE AND R EQUIREMENTS
environmental factors such as changing street surfaces.
Based on previous research projects with industry partners Using appropriate quality measures such as the mean
[21], [19], [3], [4] and three labs on autonomous vehicle squared error (MSE) of the car’s trajectory, our simulation
modeling, we identified a precise set of requirements for a framework allows for an easy comparison between different
simulation framework: (R1) Import and reuse of existing real C&C controllers.
world environment data. (R2) Capability to simulate large-
scale everyday scenarios, e.g., different traffic densities, light III. E XISTING S IMULATION E NVIRONMENTS
and weather conditions. (R3) Support for realistic and exten- Simulation frameworks represent a consistently used tool in
sible car models with sensors, controllers and actuators. (R4) the software development and testing process of self-driving
Multi-platform and portable devices support. (R5) Automated vehicles. A wide variety of frameworks already exists on the
support for continuous integration and regression testing. (R6) market. In this section, the most popular ones will be compared
Simulator should contain a physics engine. (R7) 3D visualiza- in terms of their functionalities and capability to satisfy the
tion for demonstration purposes. These requirements must be requirements presented in section II. In particular, we will
fulfilled by a versatile autonomous driving simulator in order focus on Pelops [5], Prescan [25], Mathworks Simulink 3D
to execute and validate previously developed C&C vehicles Animations [18], RTMaps [8], Gazebo [16] [10], SUMO [2]
models. [11], PTV Vissim [6], SimTram [12] [23], Vires VTD Virtual
In the remainder of this section, the usage workflow of Test Drive [20], the Udacity simulator [22], OpenDaVinci [9],
our proposed simulation framework is demonstrated based on and CarSim [17]. Finally, their differences will be summarized
a small realistic example, where two cars follow a straight with respect to the requirements.
street; videos showing our simulation framework in action are (R1) Import and reuse of existing real world envi-
available at: https://youtu.be/TwMtA6ZV86I, https://youtu.be/ ronment data: Pelops [5], a framework developed by the
d-EEs62-rGM, and https://youtu.be/HiaHf0dd_1w. Forschungsgesellschaft Kraftfahrwesen mbH Aachen, does not
(1) First, the map of interest is downloaded from present this function as a default feature. The same holds for
openstreetmap.org; based on this information the simulator Mathworks Simulink 3D Animations, RTMaps, Gazebo and
accurately recreates streets, buildings and traffic signs. (2) the Udacity simulator. Gazebo, which is a widely used simula-
Next, a simulation model as shown in Figure 1 is created. tion framework allows the design of complex maps, but not the
It contains an environment description as well as simula- import of OpenStreetMap data, which can only be performed
tion parameters such as location, time, resolution, weather by using external modules. Conversely, Prescan, SUMO, PTV
conditions, and available vehicles. (3) New vehicle models Vissim, and SimTram provide data import functionality from
can be created as shown in Figure 2 if needed. Here, one OpenStreetMap.
specific car is assembled by assigning its physical and visual (R2) Capability to simulate large-scale everyday scenar-
properties as well as its controlling unit, sensors and actuators. ios: This is a typical feature of high-level traffic simulators
For each input port of the controller, our framework delivers which allows, for instance, to analyze different traffic densities
three default virtual sensors with different noise models. (4) or different weather conditions. PTV Vissim [6], SUMO, and
The logic of the self-driving vehicle is modeled as a C&C SimTram are typically used for large-scale simulations [23].
controller mapping the sensor inputs to actuator commands. Mathworks Simulink 3D Animation, Gazebo, and CarSim
Therefore, we use the MontiCAR C&C modeling language offer limited support for such simulations, i.e., they are not
family together with its embedded math expression language optimized for large-scale scenarios. Pelops, RTMaps, and the
for behavior definitions [13]. Figure 4 depicts a simple lane Udacity simulator do not allow simulations of this kind due
keeping control system for straight streets (road curvature = to their focus on a single vehicle.
0 ◦ ) inspired by [15]; due to better readability, the body of (R3) Support for realistic and extensible car models
the LaneKeepingController is represented graphically with sensors, controllers and actuators: Being able to
in Figure 5 instead of showing the textual syntax. The input simulate different car, sensor, and controller models is a core
port of the lane keeping control system d[4] represents a feature of our framework. Among the considered simulation
distance array to the road markers (see Figure 5 for a detailed frameworks, only SUMO, PTV Vissim and SimTram do not
explanation); the output port s represents the desired steering offer this feature due to their pure high-level nature. [16] and
angle of the wheels. The controller contains a simple cor- [9] display how different models are implemented in Gazebo
rection component to automatically adapt steering errors due and OpenDaVinci, respectively. [16] explains the modeling of
to imperfect sensor inputs or actuator delays (see difference an all-wheel driver car with GPS and LIDAR whose dynamics
in Measured car state (at t=1s) and Calculated car state for is handled by the Open Dynamics Engine. [9] demonstrates
t=1s (at t=0s) in Figure 5). (5) Finally, new sensors and how to implement a lane detection application by using the
actuator models can be created. Figure 3 shows a simple OpenDaVinci framework.
1 simulation Example1 { Sim 1 car AC-SE001 { Car
1 component interface Act EMA
2 map = AachenCity.osm; 2 dimension = 4.43m,1.93m,1.25m; 2 { ports in T wVal,
3 startTime = 22.06.2017 13:30; 3 visualModel = R8Red.json; 3 out T aVal; }
4 deltaT = 1ms 4 weight = 1‘655 kg; 4 component StrgFIR4(Q(0:1)^5 h)
5 weather = noRain, noSnow; 5 controller = 5 implements Act {
6 duration = 30s; 6 LaneKeepingController; 6 ocl inv coef: sum(h) == 1;
Start Position Controller‘s input port 7 implementation Math {
7 cars { 7 sensors {
8 AC-SE001:50°46'43.7"N 6°03'38.6"E -> 8 SpeedSensor => velocity; 8 static Z(-45°:45°)^4 d =
9 50°46'49.7"N 6°04'32.5"E, 9 TiHighAccGPS => position; 9 zeros(1,4);
10 M-SE003: … -> … }} Target Position 10 Compass => direction; } 10 aVal =wVal*h(1)+ d*h(2:end);
11 actuators { Controller‘s output port 11 d = [wVal d(1:end-1)];
Fig. 1: Example Simulation Model. 12 x steering => SteeringFIR4; }} 12 aVal = saturate(aVal +
13 normrnd(0, 0.01),
14 -45°, 45°); } }
Fig. 2: Example Car Assembling Model.
Act=Actuator,Strg=Steering,w=wished,a=actual,val=value
EMA EMA
component LaneKeepingController { component Sensors2CarState Fig. 3: Steering actuator.
ports in Q(0m:0.1m:5m) d[4], (Q(0m:20m) L, Q(0m:2.4m) w) {
out Q(-45°:0.2°:45°) s; ports in Q(0m:0.1m:5m) d[4],
s out Q(-2.5m:2.5m) y, °
Q(0:360°) phi; 2.88 ⋅
CalculateNext
Right road marker
SteeringAngle
implementation Math { Maps distance to steering angle
d[1] y phi=0.5(asind(((d(2)-d(1))/L)+
Sensors2
CarState
d[2] s s : → ° asind(((d(3)-d(4))/L));
CarState2
+ calcPhi
CarState
phi Steering of wheel sin )+
d[3] y=saturate(
+ d1+ L/2*sin(phi)+ sin ))
d[4] w
calcY w/2*cos(phi), d3
-2.5m, 2.5m); } } d1
Measured car state (at t=0s)
L
EMA d2 1 ⋅ sin
component CarState2SteeringAngle Measured car d4 2
deltaS state (at t=1s) ⋅ cos
{ ports in Q(-2.5m:0.1m:2.5m) y,
calcY Calculated car state y
Correct Q(0:0.1:360°) phi, (x=0, y=0)
out Q(-45°:0.02°:45°) s; for t=1s (at t=0s)
Steering
implementation Math {
Angle const a = -2.88°/m^3;
calcPhi Fig. 5: Explanation for C&C
s=saturate(phi + a*y^3,
} -45°, 45°); } } LaneKeepingController.
Fig. 4: Overview of C&C Controller models.
(R4) Multi-platform and portable devices support: Car- sented frameworks including ours.
Sim, PTV Vissim, and Prescan exclusively support Windows
machines. All other frameworks support Linux, as well. Math- IV. F RAMEWORK FOR THE S IMULATION OF C&C MODELS
works Simulink 3D Animation supports also a 3D animation IN REALISTIC ENVIRONMENTS
web viewer based on HTML5.
An overview of the proposed MontiSim framework is shown
(R5) Automated support for continuous integration and
in Figure 6. The main features provided include the support for
regression testing: This requirement is only fulfilled by
rigid body based physics simulation, computer vision, versatile
Gazebo and the OpenDaVinci framework. Gazebo runs a
sensors, control systems, actuators, car models, real environ-
variety of regression tests concerning both the simulation and
ment data as well as continuous integration and regression
the physical robot either on an own cluster or in its Hudson
testing. The framework is based on the discrete time paradigm,
regression test suite. OpenDaVinci itself is developed using
i.e., the state of the simulation is updated in predefined discrete
continuous integration on Jenkins. Furthermore, its modular
time steps. Furthermore, the main simulator can be coupled
structure facilitates unit testing of new self-driving vehicle
with an arbitrary amount of helper simulators in order to add
algorithms.
specific capabilities such as vehicle to vehicle communication
(R6) Physics engine: While Pelops, Prescan, Gazebo or tire pressure simulation. Helper simulators can be both
and CarSim provide their own physics engines, Mathworks discrete time or discrete event based.
Simulink 3D Animation does not include out-of-the-box The main part contains one or multiple car models describ-
physics. However, it allows to import a car modeled in ing a particular car and one or multiple MontiCar C&C models
Simulink and to simulate it. The Udacity simulator, RTMaps, to model control systems. These models are independently de-
SUMO, PTV Vissim, OpenDavinci, and SimTram do not veloped by the vehicle engineer and the controller developer.
provide a real physics engine. A test case designer creates one or multiple simulation models
(R7) 3D visualization for demonstration purposes: In the describing the simulation. Furthermore, additional environment
context of the considered frameworks, SUMO and SimTram data can be inserted. All sensors and actuators are modeled
are the only frameworks not supporting a 3D visualization. and integrated by a domain expert. Apart from the MontiCar
Table I summarizes the comparison between all the pre- C&C model, each model is directly inserted into the simulation
TestingSensors
MontiCar
Car Simulation
3D Model
Assembler Visualization Optimal
1 stream StearingAngleTest Stream
3D Designer 2 for Sensors2CarState Deviation: path
Values for input port d (L=3m, w=2m) { 12,5%
Actuator
Framework
Actuator
Car Model Actuator
MontiSim
3 d = [50cm 50cm 2.5m 2.5m]
C&C to C++ 4 tick [50cm 1.3m 2.6m 1.8m];
MontiCar
Generator Expected Values for output ports time step
Actuator
Actuator
Sensor
5 phi = 0° tick 15.45°+/-0.05°;
Deviation:
C&C Model 22,5%
6 y = - tick -; 15.4° ≤ exp.
7 }Do not care about value value ≤ 15.5°
Road
Simulation
marker
Controller Model
Developer Test Case Actuator
Environment
Actuator Domain Fig. 7: Stream Test Model.
Designer Data Expert Fig. 8: Deviation of Sensor
Fig. 6: Simulation Framework and its actors.
√
TABLE I: Comparison of simulation frameworks, : yes, P: A. Translation of C&C models to C++ code
partially, -: no
One main requirement for the MontiCAR C++ Code gen-
erator is to be compatible with the widely used Simulink. If
Physics Engine Support (R6)
we transform Simulink block diagrams into MontiCAR C&C
Multiplatform Support (R4)
models, the MontiCAR model will produce for the same input
Environment Data (R1)
Support for Extendible
Automatic Continuous
3D Visualization (R7)
Simulate Large Scale
values the same output values as the original Simulink model.
Import and Reuse
This allows us to integrate Simulink models in our toolchain.
Car Models (R3)
Integration (R5)
Scenarios (R2)
In the first step, our C++ code generator calculates the exe-
Frameworks
cution order of the C&C component instances. This creates a
Simulator
sorted execution list, equivalent to Simulink slist command,
where only atomic components are considered. All other C&C
√ √ √ Win/Lin/ √ √ √ components are similar to the Simulink virtual subsystems
MontiSim
√ Mac/Web √ √ and are flattened so that only their atomic components are
Pelops √- √- √ - - √ √ considered. The calculation of the execution order, which
Prescan Win -
√ √ Win/Lin/ √ is mostly not unique, satisfies the two main rules: (i) If
Mathworks - - P
√ Mac/Web √ component C1 is connected with component C2, i.e, there
RT Maps - √- √ Win/Lin -
√ -
√ √ is at least one connector from any outgoing port of C1 to
Gazebo √- √ Win/Lin
SUMO √ √ - Win/Lin - - - any ingoing port of C2, than the execution order of C2 must
√
PTV Vissim √ √ - Win - - √ be greater than the execution order of C1. (ii) Components
SimTram √ -
√ Win/Lin - -
√ √ having no input ports (e.g. constant components) can have
Vires VTD - √ Lin - √ any execution order as long as they satisfy (i).
Udacity - √- √ Win/Lin/Mac -
√ - √
OpenDaVinci - √ √ Win/Lin √ -
√ √ The C++ code generator now uses the execution order to
CarSim - Win invoke the calculation methods of the atomic blocks in the
right order. In this context, the input values are pointers of the
previously executed calculation method results so that objects
framework. The MontiCar C&C model is translated to multi- do not need to be copied.
threaded C++ code, which is executed to control a vehicle In MontiCAR, the behavior of atomic blocks is described
during the simulation. via MontiMath, a strongly typed Matlab-like Domain Specific
Language (DSL). To enable high-speed matrix operations, the
A big advantage of the presented simulation framework
body of the calculation methods invokes the highly optimized
is the fact that all C&C languages can relatively easily be
Octave [7] C++ implementation using the Intel Math Kernel
translated into another C&C language. Therefore, we will
Library [24].
from now on refer to only one representative C&C language,
To tackle the issue that one atomic component A can
MontiCar, which we developed.
produce an output x in km/h and the other one B wants
The visualization part of the simulation framework consists input values in m/s, the C++ code generator converts all input
of 3D models, which are created by a 3D designer and used and output values to default SI unit values. In this example
in the visualization of the simulation. At the current state of x is divided by 3.6 at end of the calculation method before
development, it is possible to fluently simulate 2D scenarios being transferred to B. Thereby, the modeler benefits from
with up to 50 cars and 3D scenarios with up to 5 cars. SI unit support including generator errors, e.g., when trying
In the following, each element of the simulation framework to connect kg with m/s, as well as optimized Basic Linear
is explained in detail. Algebra Subprograms (BLAS) libraries.
B. Provide Realistic Environment JScience library. In this way, unit incompatible computations
result in Java compile errors.
The simulation of the environment relies upon Open-
StreetMap which is a 2D model. Thus the third dimension The simulator (1) executes the sensors models, than (2) it
has to be obtained from a different source or needs to be transforms all JScience values to standard SI values and finally
sampled as a random variable from an appropriate probability it (3) invokes the generated C++ code by JNI. Afterwards, the
distribution. unitless C++ code result values of the large C&C model are (4)
translated back to JScience values. Subsequently, the (5) Java
In the generated map, for each intersection road signs
actuators are executed and, eventually, (6) the physics engine
and traffic lights are randomly generated, as well. Similarly,
updates all objects (vehicles, pedestrians) in the world. When
pedestrians can be included. Their behavior is defined by a
this process is completed, everything starts from (1) again.
set of five parameters which define their movement between
two defined points P0 and P1: (1) the distance the pedestrian
moves during each time frame (2) the side of the road on D. Fluent Visualisation of Simulated Data
which the pedestrian is (3) the direction, either from P0 to P1 For systems or acceptance tests, it is helpful to visualize the
or the opposite (4) whether the pedestrian will cross the road simulated data. This way, it is possible to figure out missing
or not (5) the instantaneous position of the pedestrian. The unit or integration tests, such as "the car should not to leave
weather conditions can be either fixed, constantly or randomly the road", "the car should not wobble too much", "the car
changing. should not park too close in parking slot, otherwise the door
Vehicles have access to the simulated environment cannot open".
via sensors implementing the Sensor interface; avail- Another important reason that justifies the visualization is
able sensor classes include, e.g., a WeatherSensor, the test case designer’s motivation, i.e., they produce models
LocationSensor, CameraSensor, CompassSensor, of better quality when the executed result can be seen. A
SpeedSensor, SteeringAngleSensor, Distance- visualization allows to demonstrate their models’ features, to
ToRightSensor. Adding or modifying sensors can be done actually see the performance of their automated vehicles and
easily. to better compare different models in order to identify the best
one.
C. Aggregation of C++ Code and Environment in Simulator The visualization is realized as follows. The server sends
/ Physics Engine the states of the objects (cars and pedestrians) to the ThreeJS
The actuator, as shown in Figure 3, can either be written library via web sockets for rendering the world. In order to
in Java or in MontiMath. Since the Actuator interface have a fluent visualization, the JavaScript visualization script
for MontiMath has one input and one output port of the interpolates the car states, so that it can fluently drive between
same type, the simulator engine generates one large C&C two discrete state updates preventing the car from hopping
model, which has the controller and all MontiMath actuators from one place to another. Technically, the visualization con-
as child components. According to the car assembly model, the sists of two parts, an init to create the world and a loop
controller output ports are connected with the actuator inputs. part to move the objects.
All input ports and unused output ports of the controller are a) Init.: The simulator delivers all data required to build
directly connected to the large C&C model, also all output up the entire map at the beginning, which contains (i) the
ports of the actuators are connected to the corresponding terrain as bounds and height-map data, (ii) the street as 4D
output ports of the large C&C model. Due to the Actuator array (x,y,z position and street width) of nodes, (iii) the street
interface, the large C&C model has always the same interface signs, delivered as a JSON file for the geometry, a PNG image
as the controller. The large C&C model is now translated to file as well as a 3D position point in the map and a 3D
C++ code in order to take advantage of the BLAS operations. vector telling in what direction the sign points, (iv) the traffic
Note that the simulator, the sensors and actuators are written lights, handled similarly to the street signs. To make the scenes
in Java. more realistic, the visualization supports a day and night cycle
The generic Java Sensor interface contains one method and three different light sources: ambient light (that globally
that has two pointers as parameters, one to the complete illuminates all objects in the scene), directional light (that gets
world object and one to the current car object. This method emitted in a specific direction), hemisphere light (positioned
returns one calculated value. Based on the external simulated directly above the scene).
environment and objects, the sensor can calculate an output. b) Loop.: In the loop, the visualization sets the position
The generic Java Actuator interface contains also one of the cars and the pedestrians. In order to have a fluent
method with a HashMap containing all input and output visualization at 60 fps, which is independent from the steps per
values of the C&C controller, e.g., the engine actuator for the second that can be calculated by the simulator, the visualiza-
acceleration does not only depend on the target acceleration tion has its own simple interpolation algorithms for positions
but also on the current speed. In contrast to the generated C++ and directions of all moved objects. This way the simulation
code, where all units are compatible by construction and thus looks fluent even if a position must be retransmitted due to
ignored, the Java code for sensors and actuators works with the data loss.
Since the simulator only considers the friction coefficient of any value between 15.4 ◦ and 15.5 ◦ is accepted. Finally, the
the road, it does not model rain drops to save computational expected output values for y, i.e., the relative y-position of
power on the server. Thus, the JavaScript web client contains a the car are defined in l.6. Since these values are irrelevant for
rain controller that creates a specified amount of rain or snow this test, both time steps are defined by dashes.
particles (which may have independent sizes) randomly in the If a unit test is successful, the result is a list of components
world and moves the existing particles around, e.g., by just used in this test case. Based on this information, a test
decrementing the y coordinate. coverage is computed. If a unit test fails, then the list of
failing components is presented. Note that the inner-defined
V. C&C U NIT, I NTEGRATION , AND S YSTEM T ESTING C&C components are regarded, as well; testing a large C&C
Apart from the simulation, MontiSim addresses testing component with many inner subcomponents is mostly called
purposes in order to validate sensors, controllers, actuators, black-box integration testing.
as well as other components. Hence, in the remainder of this
section, two testing methods are presented. First, a unit testing B. C&C Integration, and System Testing with Environment
method without environment is presented. Second, a C&C Unit testing controllers, sensors, or actuators can also be
integration test method is explained to test controllers with done in an environment to simulate real world scenarios. For
the environment. example, consider the example in Figure 8. Here the car
A. C&C Unit, and Integration Testing without Environment should follow the straight line autonomously. However, the
path computed by the controller is not ideal as shown in the
The aim of C&C black-box unit testing, similar to JUnit figure.
testing, is to validate the correctness of single C&C compo- To test different controllers, the deviation from the op-
nents by testing the component with given time-dependent in- timal path can be computed for each controller. Based on
put values and comparing its calculated time-dependent output this deviation, the most optimal controller can be identified.
values with the specified ones given in the stream test file. This In this testing scenario, the deviations are totalized and
methodology provides an easy and intuitive way of testing compared. Starting from an ideal position on the line, the
C&C components. To perform, e.g., in Simulink, a similar environment is initialized (void init(World w)). After
unit testing for subsystems (analogue to C&C components) each simulation time step (void simulateTick(World
the tester needs to execute the following steps for each test w)), the deviation from the ideal position is added to the
case: (1) create a new Simulink model as a new slx file, (2) previous deviation. When the simulation is finished (void
import the subsystem to be tested as model reference, (3) add finish(World w)), the total deviation is computed as
two signal builder blocks and manually specify (or via Matlab
deviation[m]
script) the time-dependent input values and the expected output s = and it is checked whether the
driven distance[m]
values, (4) manually connect all output signals of the first simulation was successful (boolean success()).
signal builder to the input values of the model reference System tests including the environment enable model-in-
block, (5) compare each output of the model reference with the-loop testing of liveness, smoothness, and responsiveness
the expected values of the second signal builder block by of discretized closed-loop controllers [14].
connecting both to the input ports of an equal subsystem, (6)
connect the output ports of all equal blocks with the Simulink VI. C ONCLUSION
stop block, so that the simulation and thus the test will fail
Simulating self-driving vehicles is essential but still a highly
if the calculated output and the expected output are not the
complex task. Hence, in this paper, we presented a simulation
same. Our experience in several industry projects as well as
framework that aims to address extension and adaptation con-
in student labs showed that, due to the high technical obstacles,
cerns, real-world environments as well as real-world sensors
nearly no unit testing of subsystems is done in Simulink; and
and actuators. In particular, it provides an integration of high-
even then no automatic regression testing on build servers
level and low-level simulation, which allows to analyze large-
is done. As a consequence, errors are only detected later
scale scenarios as well as to address low-level details, e.g.,
by inspecting the output values manually in a time-intensive
the steering angle of the wheels. Furthermore, the framework
testing phase.
supports integration and unit testing with and without the
This paragraph now explains how the C&C steering con-
environment, respectively. This can be used to test sensors,
troller (Sensors2CarState component (s. l. 2)) can be
actuators, and controllers. Based on a small scale example,
tested in our simulation framework. For this controller, Fig-
we have demonstrated the applicability of the framework.
ure 7 shows a test case for testing the correct calculation
of the steering angle (phi) of the Sensors2CarState Acknowledgements This research was supported by a Grant from the GIF, the German-
component. In this example, the input values for which the Israeli Foundation for Scientific Research and Development, and by the Grant SPP1835
from DFG, the German Research Foundation.
component is tested are defined in ll.3-4. Note that the tick
represents one time step, e.g., going from t = 0s to t = 0.1s.
R EFERENCES
The expected output values for phi, i.e., the calculated
angle, are defined in l.5. as 15.45 ◦ +/-0.05 ◦ meaning that [1] ISO 26262: Road Vehicles : Functional Safety. ISO (2011)
[2] Behrisch, M., Bieker, L., Erdmann, J., Krajzewicz, D.: Sumo – sim- [23] Viral Patel, Manish Chaturvedi, S.S.: Comparison of sumo and sim-
ulation of urban mobility an overview. Proceedings of SIMUL 2011, tram for indian traffic scenario representation. Transportation Research
The Third International Conference on Advances in System Simulation Procedia (2016)
(2011) [24] Wang, E., Zhang, Q., Shen, B., Zhang, G., Lu, X., Wu, Q., Wang,
[3] Bertram, V., Manhart, P., Plotnikov, D., Rumpe, B., Schulze, C., Y.: Intel math kernel library. In: High-Performance Computing on the
Wenckstern, M.v.: Infrastructure to Use OCL for Runtime Structural Intel R Xeon PhiTM , pp. 167–188. Springer (2014)
Compatibility Checks of Simulink Models. In: Modellierung 2016 [25] Zhizhou Wu, Jie Yang, L.H.: Study on the collision avoidance strategy
Conference. LNI, vol. 254, pp. 109–116. Bonner Köllen Verlag (March at unsignalized intersection based on prescan simulation. 13th COTA
2016), http://www.se-rwth.de/publications/Infrastructure-to-Use-OCL- International Conference of Transportation Professional (2013)
for-Runtime-Structural-Compatibility-Checks-of-Simulink-Models.pdf
[4] Bertram, V., Maoz, S., Ringert, J.O., Rumpe, B., von Wenckstern, M.:
Component and Connector Views in Practice: An Experience Report.
In: MoDELS (2017)
[5] Forschungsgesellschaft Kraftfahrwesen mbH, A.: Pelops, white paper
http://www.pelops.de
[6] Gabriel Gome, Adolf May, R.H.: Congested freeway microsimulation
model using vissim. Transportation Research Record: Journal of the
Transportation Research Board (2004)
[7] Hansen, J.S.: GNU Octave: Beginner’s Guide: Become a Proficient
Octave User by Learning this High-level Scientific Numerical Tool from
the Ground Up. Packt Publishing Ltd (2011)
[8] I. Abuhadrous, F. Nashashibi, C.L.: Multi-sensor data fusion for land
vehicle localization using /sup rt/maps. Intelligent Vehicles Symposium.
Proceedings (2003)
[9] Ibtissam Karouach, S.I.: Lane detection and following approach in
self-driving miniature vehicle. Bachelor of Science Thesis in Software
Engineering and Management, University of Gothenburg (2016)
[10] Koenig, N., Howard, A.: Design and use paradigms for gazebo, an
open-source multi-robot simulator. In: Intelligent Robots and Systems,
2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ International Confer-
ence on. vol. 3, pp. 2149–2154. IEEE (2004)
[11] Krajzewicz, D., Erdmann, J., Behrisch, M., Bieker, L.: Recent devel-
opment and applications of sumo-simulation of urban mobility. Inter-
national Journal On Advances in Systems and Measurements 5(3&4),
128–138 (2012)
[12] Kumar Abhilash, Sarkar Partha Pratim, S.T.K.: Studying and simulating
mix traffic using simtram. Traffic Engineering & Control (2016)
[13] Kusmenko, E., Roth, A., Rumpe, B., von Wenckstern, M.: Modeling
architectures of cyber-physical systems. In: 13th European Conference
on Modelling Foundations and Applications. Springer (2017)
[14] Matinnejad, R., Nejati, S., Briand, L., Bruckmann, T., Poull, C.: Au-
tomated Model-in-the-Loop Testing of Continuous Controllers Using
Search, pp. 141–157. Springer Berlin Heidelberg, Berlin, Heidelberg
(2013)
[15] Oliver Toro, Tamas Becsi, S.A.: Design of a lane keeping algorithm of
autonomous vehicle. Periodica Polytechnica Transportation Engineering
(2015)
[16] Özgüner, Ü., Redmill, K., Biddlestone, S., Hsieh, M.F., Yazici, A.,
Charles, T.: Simulation and testing environments for the darpa urban
challenge. IEEE International Conference on Vehicular Electronics and
Safety (2008)
[17] Patil, K.S., Jagtap, V., Jadhav, S., Bhosale, A., Kedar, B.: Generating
a 3d simulationof a car accident from a written description in natural
language: the carsim system. Proceedings of the workshop on Temporal
and spatial information processing (2001)
[18] Patil, K.S., Jagtap, V., Jadhav, S., Bhosale, A., Kedar, B.: Performance
evaluation of active suspension for passenger cars using matlab. IOSR
Journal of Mechanical and Civil Engineering (2013)
[19] Richenhagen, J., Rumpe, B., Schloßer, A., Schulze, C., Thissen, K., von
Wenckstern, M.: Test-driven Semantical Similarity Analysis for Software
Product Line Extraction. In: International Systems and Software Product
Line Conference (SPLC ’16). pp. 174–183. ACM (2016)
[20] Roth, E., Dirndorfer, T.J., Knoll, A., v. Neumann-Cosel, K., Ganslmeier,
T., Kern, A., Fischer, M.O.: Analysis and validation of perception sensor
models in an integrated vehicle and environment simulation. Proceedings
of SIMUL 2011, The Third International Conference on Advances in
System Simulation (2011)
[21] Rumpe, B., Schulze, C., Wenckstern, M.v., Ringert, J.O.,
Manhart, P.: Behavioral Compatibility of Simulink Models
for Product Line Maintenance and Evolution. In: Software
Product Line Conference (SPLC’15). pp. 141–150. ACM (2015),
http://www.se-rwth.de/publications/Behavioral-Compatibility-of-
Simulink-Models-for-Product-Line-Maintenance-and-Evolution.pdf
[22] Udacity: Self-driving car simulator. GitHub (2016)