ions generalize better than a model trained directly on RGB images in simulation, even when the perception model is trained on realworld data. We also show that the perception model trained on several tasks using multi-task learning, leads to better-performing driving policies than learning only semantic segmentation.
Autonomous vehicles have been a popular research domain for many years, and there has recently been large investments from both technology and car companies to be the rst to solve the problem. The most prominent approach in recent years has been the modular approach, where the driving is divided into several sub-tasks such as perception, localization, and planning. The modular approach often results in a very complex solution, where each module has to be ne tuned individually. The scalability of this approach can therefore become an issue when expanding the approach to more complex situations.
Another rising approach is the end-to-end approach, where the entire driving policy is generated within a single system. The system takes sensor-input and converts it directly to driving commands, similar to how humans drive vehicles. End-to-end systems for autonomous vehicles require large amounts of data, and the ability to train on many di erent scenarios. Therefore, simulated environments have been explored for training in di erent scenarios and creating large datasets. These environments, however, di er signi cantly from the real world, and the learned driving policy does not transfer adequately between environments.
This paper attempts to improve the ability for driving policies to be
transferred between domains by abstracting away both the perception task, and the
raw throttle and brake control of the vehicle, focusing mainly on the perception
task. The Mapillary Vistas dataset [
The paper is structured as follows: Section 2 investigates related work, while additionally providing a brief history of the eld itself. Section 3 presents our method, including the data, neural network architectures, and evaluation metrics. Section 4 describes our experiments, their results, and discussion related to these. Section 5 discusses the overall implications of the experimental results, and compare our results with conclusions from related work. Section 6 draws a nal conclusion of the work conducted, and addresses the paper's merits, weaknesses, and potential future work. 2
Related Work
[
The end-to-end approach was rst demonstrated in the ALVINN project,
described by [
Transferring driving policies between domains also require an abstraction
of the perception data. [
Our approach consists of two separately trained models: a perception model and a driving model. The reason for this separation is to decouple the task of scene understanding from the task of driving. This opens up the possibility of improving the tasks independently, and we can train the models separately and with di erent datasets. An important goal was then to make the output of the perception model domain-independent, which in turn makes the driving policy model domain-independent. Domain independence is in this context de ned as the ability to generalize between multiple domains (e.g. simulated and real), as well as completely unseen domains.
The perception model is trained on datasets containing RGB images paired with semantic segmentation and depth information. These datasets can contain images not directly related to driving, as they are used to train a general scene understanding.
The driving model is trained on datasets recorded from an expert driver. The dataset contains RGB images and driving data such as steering angle, current speed and target speed. The datasets for both models can either be collected from the real world, generated from the CARLA simulator, or a combination of both real-world and simulated data. 3.1
The perception model takes raw RGB images as input, and tries to predict one
or more outputs related to scene understanding; always semantic segmentation,
and in some experiments an additional depth map. The model has an
encoderdecoder structure, compressing the input into a layer with few neurons (encoder)
before expanding towards one or more prediction outputs (decoders). To train
the model, data from driving situations in di erent environments and
geographical areas are used. Some experiment also use data generated from CARLA as a
means to improve the model's performance in simulated environments.
Data. The Mapillary Vistas dataset [
In addition to using real-world perception data, we generated synthesized
data in CARLA. A Python script spawns a large variety of vehicles and
pedestrians, and captures RGB, semantic segmentation, and depth data from the
vehicles as they navigate the simulated world. The eld of view (FOV) and
camera yaw angle were randomly distributed to generalize between di erent camera
setups. The simulated weather was additionally changed periodically; varying
cloudiness, amount of rain, time of day, and other modi able weather
parameters in CARLA. The nal size of this synthesized dataset is about 20 000 images.
Architecture. Several encoders and decoders were explored when deciding the
model's architecture. Encoders tested were: MobileNet [
Evaluation and Metrics. The segmentation prediction was evaluated using IntgetrissectthieongroovuenrdUtnriuotnh(sIoegUm),ecnatalctuiolanteadndwipthistthheefoplrleodwiicntgedeqsuegamtioennt:aggttti\[oppn,. wMheearne IoU was used as the main indicator for performance, calculated by taking the mean of the class-wise IoU. Frequency weighted IoU was also calculated, measured as the mean IoU weighted by the number of pixels for each class.
The accuracy within threshold, as described in [
The driving model runs raw RGB images through the perception model, and uses its output segmentation and depth predictions as input. These images are coupled with driving data recorded from an expert driver. The driving model processes these inputs through its own layers, before outputting a steering angle and target speed.
Data. The driving data was generated in CARLA version 0.9.9. This was done by making an autopilot control a car in various environments, and recording video from three forward-facing cameras, its steering angle, speed, target speed, and HLC (left, right, straight, or follow lane). The autopilot has access to the full state of the world, which includes a HD map, its own location and velocity, and the states of other vehicles and pedestrians. It uses this information to generate waypoints, which are nally fed into a PID-based controller to apply throttle, brake, and steering angle. The collected training data was unevenly distributed in regards to HLCs and steering angles, and we therefore down-sampled overrepresented values for an improved data distribution.
Various datasets were gathered for training the driving policy, all of which
were collected in Town01. These have di erent amount of complexity; steering
noise magnitude the autopilot has to account for, di erent weather conditions
and di erent light conditions. 30 641 samples were collected in total, where the
weather varied according to CARLA's 15 default weather presets. The training
data was e ectively multiplied by three, as we made two copies of each data
point, where we used the recorded image from each side camera instead of the
main camera. To adjust for a slightly modi ed camera perspective, we added
an o set of 0.05 and -0.05 in steering angle respectively for the left and right
camera variants. This technique was rst introduced by [
Traffic light state Speed limit Current speed
HLC + + +
Angle Speed
The segmentation and depth output of the perception model are
concatenated channel-wise, and resemble a RGBD (RGB + depth) image. This
representation is then run through 5 convolutional blocks, each consisting of zero
padding of 1, 2D-convolution with kernel 3, batch normalization, ReLu
activation, and nally max-pooling with pool size 2. The lter sizes are 64, 128, 256,
256, 256, respectively. The current HLC, whether the tra c light was red or not,
speed, and speed limit are concatenated with feature vectors generated from the
perception data. The last layers are a combination of fully-connected layers,
where we concatenate the HLC vector at each step, similar to [
r R ddct . Tra c violations were not included as metrics, as the scope of this paP
jRj per is mainly within completing routes without major incidents, and the models were therefore not trained to avoid such violations. The model's validation loss was also used as a rough metric for performance. By empirical observations we only picked models with validation loss / 0:03 for further evaluation. The validation loss metric was used as an initial performance estimation because the MCR evaluation was considerably more time consuming. 4
There are two main experiments conducted in this paper. The rst experiment and its sub-experiments focuses on generating the best perception model to be used when training the driving network. Model architecture, dataset variants, augmentation, and multi-task learning are parameters experimented with to increase performance. The second experiment is conducted in CARLA. This experiment assess the driving policy performance given the di erent models derived in the rst set of experiments. The generalizability of each model is tested using di erent unseen environments. Each perception model is then compared to a baseline model trained only on the CARLA dataset using Mean Completion Rate as the metric. 4.1
The perception experiments use semantic segmentation and occasionally depth estimation to generalize the driving environment when training and testing the driving models. All of the perception experiments use the same dataset for evaluation, and the results can therefore be compared across experiments. The CARLA data generated and used for evaluation consists of 4 400 images with corresponding ground truth segmentation and depth maps from Town 3-4. The Mapillary evaluation dataset is a set of 2 000 images from the original Mapillary test set.
nd the best encoder and decoder to use for the perception network. All the
encoders tested were picked because they have previously shown good results in
other papers, and were implemented in a common library by [
The three encoders showed increased performance as the complexity and size of the encoder increased. The Vanilla CNN encoder performed worst with the lowest Mean IoU score, however, it was also the fastest model during training and testing. MobileNet gave better results while keeping a lot of the speed advantage from the Vanilla CNN network. MobileNet also showed very good results, with MobileNet-U-Net displaying the best overall performance when combining scores. ResNet50 performed good as expected with a higher Mean IoU than MobileNet-U-Net, however the di erence from MobileNet-U-Net was less than
ResNet50-U-Net 0.383
Weighted IoU 0.712 0.705 0.775 0.774 0.767 0.733 expected. MobileNet was used for futher experiments as it was signi cantly faster than ResNet50.
Experiment 1-2: Training data. To improve the model further some CARLA
data was introduced to the Mapillary dataset. Augmentation was also introduced
for further improvements and better generalization. The Mapillary+CARLA
dataset consisted of 20 000 datapoints from the Mapillary dataset and 3 250
samples from Town01 and Town02 in CARLA. The dataset with only
augmented CARLA data (CARLA+Aug) used a di erent dataset of 15 000 samples
from Town 1-4, and 4 000 samples from Town 5 as validation. The results were
evaluated on Town 3-4 as Town 1-2 was used when training Mapillary+CARLA.
The augmentation included consists of among others gaussian noise, translation,
rotation, hue and saturation augmentations, and was adapted from [
Training dataset
The dataset experiment shows that including CARLA data as a component when training the perception models increases the total performance. As the model's goal is to make good predictions in both real and simulated environments, combining data from both seems to be a reasonable approach. CARLA+Aug achieves great results when evaluating on CARLA data, however the performance decreased drastically when predicting in real-world environments. Models trained on real-world data tends to generalize better to unseen simulated environments than the other way around. Incorporating some CARLA data into the real-world data in addition to augmenting the images yields the best results overall.
depth estimation decoder to the MobileNet-U-Net model. The model was trained with ground truth data generated by the Monodepth2 network using images from the Mapillary dataset, while depth maps for the CARLA data was included in the generated CARLA dataset.
Including a depth estimation decoder increases the segmentation performance
for each model. The mean increase in IoU on the CARLA test set is 8%, which
conforms with the results reported by [
This experiment aims to assess the overall performance of the two-part
(perception and driving policy) architecture. We run real-time evaluations on variants
of our proposed architecture, including a baseline network where the complete
network is trained at once. The evaluation is conducted with a custom
scenario runner for CARLA, originally introduced by [
The scenario runner. The scenario runner makes each model drive through a prede ned set of routes, each of which is de ned by a set of waypoints. The model navigates each route using HLCs provided automatically when passing each waypoint. Each attempt at a route ends either when the vehicle completes the route, or when the vehicle enters any of the following erroneous states: stuck on an obstacle, leaving its correct lane and not returning within ve seconds, or ignoring a HLC. The models are then compared on their mean route completion rate.
Environments and Routes. The models were tested in two environments, Town02 and Town07. Town02 is similar, but not identical to the one in the driving policy's training data, which is Town01. Town07 is quite di erent, and is rural with narrow roads (some without any centre marking), elds, and barns. There are three routes in each environment, which the models will try to complete in six di erent weather conditions. Three of the weather conditions have already been observed in the training data, while the three remaining are unknown to the policy. The training data only contain samples from day-time weathers, but two of the unknown weathers are at midnight.
Results. Table 4 summarizes the driving performance of the di erent models. The model trained only on driving data and without a frozen perception model, RGB, was the best-performing model on Town02, but it struggles with Town07.
The model names starting with SD indicates that they use the segmentation and depth perception model (henceforth SD). SD-CARLA, which uses SD trained only on perception data from CARLA, outperforms all other models when ranked by Mean Completion Rate (MCR) over both towns. To demonstrate its performance, we made a video (https://youtu.be/HL5LStDe7wY) showing some of its good performing moments. SD-Mapillary uses SD as well, but only had perception training data from Mapillary. While not performing as good as SD-CARLA, it still has impressive results. Its perception model has not seen any CARLA data, but is still able to predict segmentation and depth good enough for the driving model to beat even the RGB model. SD-Combined used perception data from both Mapillary and CARLA, and performs a little bit worse than SD-Mapillary.
The model names starting with S indicates that they use the segmentationonly perception model (henceforth S). S-CARLA is the S-counterpart of SDCARLA, and it performs very well in Town02. In Town07 however, it struggles
Seen weather Model Clear (D) Rain (D) Wet (S) RGB 100.00 % 28.72 % 36.05 % SD-CARLA 100.00 % 43.30 % 55.36 % S-CARLA 93.83 % 7.23 % 43.51 % SD-Mapillary 88.51 % 42.16 % 67.22 % SD-Combined 93.53 % 9.92 % 47.42 % S-Combined 90.71 % 44.02 % 23.12 % S-Mapillary 72.12 % 43.30 % 40.85 %
Unseen weather with night-time weather. S-Mapillary is the S-counterpart of SD-Mapillary, and it has the lowest MCR in both towns. In any run with Fog (S), it fails almost immediately. S-Combined uses combined perception data, the same as SD-Combined. It is performing a bit better than S-Mapillary in Town02, and is the fourth best in Town07. 5
Results in comparison to related work
[
[
The driving model by [
We nd that models with a learned understanding of both the semantics and/or geometry of the scene are able to navigate never-before-seen environments and weather. Our real-time experiment shows that these driving models often perform better than learning from raw image inputs directly, with models utilizing both semantics and geometry performing best overall.
Variance. It is important to note that we observed a high variance when
training and evaluating our models. Two models trained from the exact same setup
could perform signi cantly di erent, despite having the exact same training data.
We suspect that this is the same variance problem as [
Splitting end-to-end models for autonomous vehicles into separate models for perception and driving policy is shown to give good results in simulated environments. Perception models trained from public datasets such as Mapillary Vistas can be used to reduce the amount of driving data needed when training an end-to-end driving policy network. This approach opens up for training the driving policy in a simulated environment, while still getting good performance in real-world environments.
Future work should explore how these results transfers into the real world. Evaluating the performance of a model trained solely in simulation directly in a real-world environment will be an important next step as a means of testing the validity of these results.