Autonomous Driving in Simulation using Domain-Independent Perception Audun Wigum Arbo? , Even Dalen* , and Frank Lindseth Norwegian University of Science and Technology, Trondheim, Norway audun.wa@gmail.com, even.dalen@live.no, frankl@ntnu.no Abstract. There have been great advancements within the fields of computer vision and autonomous driving in recent years. Autonomous vehicle systems have typically consisted of mostly handmade algorithms and rules, where neural networks have been assisting in perception tasks. Increased usage of neural network have shown promising results, and many state-of-the-art systems now utilize these for increasingly complex tasks. This paper builds upon recent work in building end-to-end networks for autonomous driving. Conditional imitation learning is combined with a perception network that generate high level abstractions from RGB images. We examine the driving performance implications of learning to drive from raw RGB images, semantic segmentation, and depth es- timates. Secondly, we propose a driving network which uses raw RGB images to predict semantic segmentation and depth maps of the scene, which furthermore is used to predict output a steering angle and a target speed for the vehicle. The models are evaluated in CARLA, an open- source simulator for autonomous driving research, in various environ- ments: an urban town, a rural road with surrounding farms and fields, and in greatly varied weather conditions. Our experiments show that the driving network trained on higher-level abstractions generalize better than a model trained directly on RGB images in simulation, even when the perception model is trained on real- world data. We also show that the perception model trained on several tasks using multi-task learning, leads to better-performing driving poli- cies than learning only semantic segmentation. Keywords: End-to-end Autonomous Driving · AV Domain Transfer · Multi-task Learning · Conditional Imitation Learning · Semantic Seg- mentation · Depth Estimation 1 Introduction Autonomous vehicles have been a popular research domain for many years, and there has recently been large investments from both technology and car com- ? These authors contributed equally to this work. Copyright c 2020 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). Colour and Visual Com- puting Symposium 2020, Gjøvik, Norway, September 16-17, 2020. 2 A.W. Arbo et al. panies to be the first to solve the problem. The most prominent approach in recent years has been the modular approach, where the driving is divided into several sub-tasks such as perception, localization, and planning. The modular approach often results in a very complex solution, where each module has to be fine tuned individually. The scalability of this approach can therefore become an issue when expanding the approach to more complex situations. Another rising approach is the end-to-end approach, where the entire driving policy is generated within a single system. The system takes sensor-input and converts it directly to driving commands, similar to how humans drive vehicles. End-to-end systems for autonomous vehicles require large amounts of data, and the ability to train on many different scenarios. Therefore, simulated environ- ments have been explored for training in different scenarios and creating large datasets. These environments, however, differ significantly from the real world, and the learned driving policy does not transfer adequately between environ- ments. This paper attempts to improve the ability for driving policies to be trans- ferred between domains by abstracting away both the perception task, and the raw throttle and brake control of the vehicle, focusing mainly on the perception task. The Mapillary Vistas dataset [21] is used for learning perception in a real- world driving environment, and the autonomous vehicle simulator CARLA [9] is used to learn both driving and perception. The ultimate goal of this paper is to reduce the amount of real-world data required to train an autonomous vehicle, by utilizing simulated environments for training. The paper is structured as follows: Section 2 investigates related work, while additionally providing a brief history of the field itself. Section 3 presents our method, including the data, neural network architectures, and evaluation met- rics. Section 4 describes our experiments, their results, and discussion related to these. Section 5 discusses the overall implications of the experimental results, and compare our results with conclusions from related work. Section 6 draws a final conclusion of the work conducted, and addresses the paper’s merits, weaknesses, and potential future work. 2 Related Work [27] arrange autonomous vehicle control algorithms into two categories: mod- ular approaches and end-to-end approaches. Modular approaches divides the responsibility of driving into several sub-tasks, such as perception, localization, planning, and control. Conversely, end-to-end approaches can be defined as a function f (x) = a where x is any input needed to make decisions — typically sensor data and environmental information — and a are the output controls that are sent to the vehicle’s actuators. The end-to-end approach was first demonstrated in the ALVINN project, described by [22]. ALVINN was able to follow simple tracks, but had no means to handle more complex environments. Since then, large advancements have been done within neural networks, resulting in new research within end-to-end Domain-Independent Perception in Autonomous Driving 3 vehicle control. [3] approaches the problem using modern techniques, and show- case a driving policy capable of driving on both highways and residential roads; in varied weather conditions. More recent approaches [6,20,15,14,26] are based on Conditional Imitation Learning (CIL), introduced by [6] in 2017, where the driving policy is given instructions — high-level commands (HLCs) — on which actions to take (e.g. turn left in next intersection). [6] shows that an architecture can be re-used for both simulated and physical environments, but they make no attempt to use the same model weights across the two domains. Codevilla et. al outputs a steering angle, and either throttle or brake, which are sent to the vehicle’s control systems. [15] outputs the target speed of the vehicle, leaving the raw throttle and brake adjustments to a lower-level system. [20] proposes to abstract the commands even further; into several waypoints in space. Their model outputs two waypoints, 5 and 20 meters away from the vehicle, which a PID controller uses to control the vehicle’s steering and velocity. Transfer from simulation to real world. A lot of studies have been done on transferring learning from simulation to the real world. [18] used images from the driving game Grand Theft Auto, to train their object detection model, and achieved state of the art performance on the KITTI [10] and Cityscapes [8] datasets. [4] successfully used simulation to train a model for robotic grasping of physical unseen objects. Among the techniques used was applying randomization in the form of random textures, lighting, and camera position, to enable their model to generalize from the simulated source domain to their physical target domain. Transferring driving policies between domains also require an abstraction of the perception data. [20] uses a perception model to generate segmentation maps which are forwarded to the driving model, in order to generate similar per- ception environments for both simulation and real-world. [26] combines ground- truth segmentation and depth data from CARLA to increase driving perfor- mance. [15] uses an encoder-decoder network with three decoder-heads — seg- mentation, depth estimation and original RGB reproduction — to maximize the model’s scene understanding. Hawke et al. also removes the decoding-process when training their driving policy, making their driving policy model take only the compressed encoding of scene understanding as input. [19] finds that the performance of such multi-task prediction models depend highly on the relative weighting between each task’s loss. Tuning these weights manually is an error prone and time-consuming process, and they therefore suggest a solution for tun- ing weights based on the homoscedastic uncertainty of each task. They show that the multi-task approach outperformes separate models trained individually. The uncertainty based weighing was later used by Hawke et al. and produced good results for generating optimal encoding of a driving scene. Depth images has also been proven as a useful approach in other simulation-to-real world knowledge transfers, such as robotic grasping [25,12]. 4 A.W. Arbo et al. 3 Data and Methods Our approach consists of two separately trained models: a perception model and a driving model. The reason for this separation is to decouple the task of scene understanding from the task of driving. This opens up the possibility of improving the tasks independently, and we can train the models separately and with different datasets. An important goal was then to make the output of the perception model domain-independent, which in turn makes the driving policy model domain-independent. Domain independence is in this context defined as the ability to generalize between multiple domains (e.g. simulated and real), as well as completely unseen domains. The perception model is trained on datasets containing RGB images paired with semantic segmentation and depth information. These datasets can contain images not directly related to driving, as they are used to train a general scene understanding. The driving model is trained on datasets recorded from an expert driver. The dataset contains RGB images and driving data such as steering angle, current speed and target speed. The datasets for both models can either be collected from the real world, generated from the CARLA simulator, or a combination of both real-world and simulated data. 3.1 Perception Model The perception model takes raw RGB images as input, and tries to predict one or more outputs related to scene understanding; always semantic segmentation, and in some experiments an additional depth map. The model has an encoder- decoder structure, compressing the input into a layer with few neurons (encoder) before expanding towards one or more prediction outputs (decoders). To train the model, data from driving situations in different environments and geograph- ical areas are used. Some experiment also use data generated from CARLA as a means to improve the model’s performance in simulated environments. Data. The Mapillary Vistas dataset [21] (henceforth Mapillary) was used for RGB and ground-truth semantic segmentation data. The dataset consists of 25 000 high-resolution images from different driving situations, with a large variety of weather and geographical locations. To simplify the environment for the perception network, the number of classes for segmentation was reduced from the original 66 object classes, to five classes: unlabeled, road, lane markings, humans and vehicles. To train the model’s depth decoder, ground truth depth maps were generated using the Monodepth2 network from [11], as Mapillary lacks this information. Figure 1 shows a sample from this dataset. Domain-Independent Perception in Autonomous Driving 5 Fig. 1: Sample of the data used when training the perception model. The left image is the original RGB. The center image is segmentation ground truth from Mapillary. The right depth map was generated from RGB images with the Monodepth2 network. In addition to using real-world perception data, we generated synthesized data in CARLA. A Python script spawns a large variety of vehicles and pedes- trians, and captures RGB, semantic segmentation, and depth data from the vehicles as they navigate the simulated world. The field of view (FOV) and cam- era yaw angle were randomly distributed to generalize between different camera setups. The simulated weather was additionally changed periodically; varying cloudiness, amount of rain, time of day, and other modifiable weather parame- ters in CARLA. The final size of this synthesized dataset is about 20 000 images. Architecture. Several encoders and decoders were explored when deciding the model’s architecture. Encoders tested were: MobileNet [17], ResNet-50 [16] and a vanilla CNN, while decoders tested were: U-Net[23] and SegNet[2]. To gener- ate a network that could predict both depth and segmentation estimations, we modified the existing MobileNet-U-Net architecture to include a second U-Net decoder. The decoder was modified to predict only one value per pixel, use the sigmoid activation function, and train with a regression loss function for depth estimation, adapted from [1]. Figure 2 illustrates the new MobileNet-U-Net with two decoders. Fig. 2: A simplified illustration of the perception model. The different architectures all used a variant of the encoder-decoder architecture. The figure represents the MobileNet- U-Net model with a second depth estimation decoder, where each layer in the encoder is connected to the corresponding layer in the decoder. 6 A.W. Arbo et al. Evaluation and Metrics. The segmentation prediction was evaluated using In- tersection over Union (IoU), calculated with the following equation: gt∩p gt∪p , where gt is the ground truth segmentation and p is the predicted segmentation. Mean IoU was used as the main indicator for performance, calculated by taking the mean of the class-wise IoU. Frequency weighted IoU was also calculated, mea- sured as the mean IoU weighted by the number of pixels for each class. The accuracy within threshold, as described in [5], was chosen as the metric for depth estimation. Given the predicted depth value dp and the ground truth d dp depth value dgt , the accuracy δ within threshold th is defined as max( dgt p , dgt )= δ > th. Each pixel gets labeled as true or false based on whether the pixel is within the specified threshold or not. The accuracy of an image is then calculated by taking the average of all the pixels in the image. th is a threshold that we varied between the values 1.25, 1.252 , and 1.253 , as in [5]. 3.2 Driving Model The driving model runs raw RGB images through the perception model, and uses its output segmentation and depth predictions as input. These images are coupled with driving data recorded from an expert driver. The driving model processes these inputs through its own layers, before outputting a steering angle and target speed. Data. The driving data was generated in CARLA version 0.9.9. This was done by making an autopilot control a car in various environments, and recording video from three forward-facing cameras, its steering angle, speed, target speed, and HLC (left, right, straight, or follow lane). The autopilot has access to the full state of the world, which includes a HD map, its own location and velocity, and the states of other vehicles and pedestrians. It uses this information to generate waypoints, which are finally fed into a PID-based controller to apply throttle, brake, and steering angle. The collected training data was unevenly distributed in regards to HLCs and steering angles, and we therefore down-sampled over- represented values for an improved data distribution. Various datasets were gathered for training the driving policy, all of which were collected in Town01. These have different amount of complexity; steering noise magnitude the autopilot has to account for, different weather conditions and different light conditions. 30 641 samples were collected in total, where the weather varied according to CARLA’s 15 default weather presets. The training data was effectively multiplied by three, as we made two copies of each data point, where we used the recorded image from each side camera instead of the main camera. To adjust for a slightly modified camera perspective, we added an offset of 0.05 and -0.05 in steering angle respectively for the left and right camera variants. This technique was first introduced by [3], and has later proved successfully in other papers [6,14]. Domain-Independent Perception in Autonomous Driving 7 Architecture. The input of the driving model is a concatenation of the output from the perception model and an information vector containing the HLC (one- hot encoded), current speed, current speed limit, and the upcoming traffic light’s state. The driving model is trained on simulation data with all the layers of the perception model frozen (that is, non-trainable) to preserve generalizability. Figure 3 shows an overview of the model. Angle + Speed + Traffic light state + Speed limit Current speed HLC Fig. 3: A simplified illustration of the driving network. The segmentation and depth maps inputs are concatenated directly from the outputs of the perception model (shown in Figure 2). The segmentation and depth output of the perception model are concate- nated channel-wise, and resemble a RGBD (RGB + depth) image. This rep- resentation is then run through 5 convolutional blocks, each consisting of zero padding of 1, 2D-convolution with kernel 3, batch normalization, ReLu activa- tion, and finally max-pooling with pool size 2. The filter sizes are 64, 128, 256, 256, 256, respectively. The current HLC, whether the traffic light was red or not, speed, and speed limit are concatenated with feature vectors generated from the perception data. The last layers are a combination of fully-connected layers, where we concatenate the HLC vector at each step, similar to [15]. The first out- put of the model is the steering prediction; one neuron outputting the optimal steering (between 0 and 1, 0 being max leftward, 1 being max rightward), later mapped to CARLA’s [-1, 1] range. The second output is the optimal vehicle speed, outputted as a percentage of 100 km/h (between 0 and 1). Evaluation and Metrics. The main metric used for measuring driving model performance was Mean Completion Rate (MCR) during real-time evaluation. This is calculated by dividing the completed distance dc by the total route distance dt of each run-through of a route, averaged over all run-throughs R: 8 A.W. Arbo et al. P dc rR dt |R| . Traffic violations were not included as metrics, as the scope of this pa- per is mainly within completing routes without major incidents, and the models were therefore not trained to avoid such violations. The model’s validation loss was also used as a rough metric for performance. By empirical observations we only picked models with validation loss / 0.03 for further evaluation. The val- idation loss metric was used as an initial performance estimation because the MCR evaluation was considerably more time consuming. 4 Experiments and results There are two main experiments conducted in this paper. The first experiment and its sub-experiments focuses on generating the best perception model to be used when training the driving network. Model architecture, dataset vari- ants, augmentation, and multi-task learning are parameters experimented with to increase performance. The second experiment is conducted in CARLA. This experiment assess the driving policy performance given the different models de- rived in the first set of experiments. The generalizability of each model is tested using different unseen environments. Each perception model is then compared to a baseline model trained only on the CARLA dataset using Mean Completion Rate as the metric. 4.1 Experiment 1: Perception Model The perception experiments use semantic segmentation and occasionally depth estimation to generalize the driving environment when training and testing the driving models. All of the perception experiments use the same dataset for evaluation, and the results can therefore be compared across experiments. The CARLA data generated and used for evaluation consists of 4 400 images with corresponding ground truth segmentation and depth maps from Town 3-4. The Mapillary evaluation dataset is a set of 2 000 images from the original Mapillary test set. Experiment 1-1: Encoder-decoder models. This experiment attempts to find the best encoder and decoder to use for the perception network. All the encoders tested were picked because they have previously shown good results in other papers, and were implemented in a common library by [13]. The three encoders showed increased performance as the complexity and size of the encoder increased. The Vanilla CNN encoder performed worst with the lowest Mean IoU score, however, it was also the fastest model during training and testing. MobileNet gave better results while keeping a lot of the speed advan- tage from the Vanilla CNN network. MobileNet also showed very good results, with MobileNet-U-Net displaying the best overall performance when combin- ing scores. ResNet50 performed good as expected with a higher Mean IoU than MobileNet-U-Net, however the difference from MobileNet-U-Net was less than Domain-Independent Perception in Autonomous Driving 9 Model Mean IoU Weighted IoU VanillaCNN-SegNet 0.324 0.712 VanillaCNN-U-Net 0.351 0.705 MobileNet-SegNet 0.368 0.775 MobileNet-U-Net 0.403 0.774 ResNet50-SegNet 0.405 0.767 ResNet50-U-Net 0.383 0.733 Table 1: Evaluation of three different encoders (Vanilla CNN, Mobilenet and ResNet50), and two decoders (SegNet and U-Net). Each model was trained on the Mapillary dataset (18 000 samples for training and 2 000 for validation) without any augmentation. The best scores are marked in bold. expected. MobileNet was used for futher experiments as it was significantly faster than ResNet50. Experiment 1-2: Training data. To improve the model further some CARLA data was introduced to the Mapillary dataset. Augmentation was also introduced for further improvements and better generalization. The Mapillary+CARLA dataset consisted of 20 000 datapoints from the Mapillary dataset and 3 250 samples from Town01 and Town02 in CARLA. The dataset with only aug- mented CARLA data (CARLA+Aug) used a different dataset of 15 000 samples from Town 1-4, and 4 000 samples from Town 5 as validation. The results were evaluated on Town 3-4 as Town 1-2 was used when training Mapillary+CARLA. The augmentation included consists of among others gaussian noise, translation, rotation, hue and saturation augmentations, and was adapted from [13]. CARLA Eval Mapillary Eval Training dataset Mean IoU Weighted IoU Mean IoU Weighted IoU Mapillary 0.425 0.771 0.632 0.887 Mapillary+Aug 0.436 0.809 0.574 0.873 Mapillary+CARLA 0.469 0.846 0.633 0.889 Mapillary+CARLA+Aug 0.478 0.850 0.568 0.874 CARLA+Aug 0.572 0.909 0.384 0.785 Table 2: Evaluation of different datasets on the MobileNet-U-Net model. Mapillary is the original dataset while the CARLA dataset was generated directly from CARLA. Each dataset consist of about 20 000 samples, and the Mapillary+CARLA dataset con- sist of about 80/20 Mapillary and CARLA data respectively. The two sets of columns show evaluation on CARLA data and Mapillary data respectively. The best scores are marked in bold. 10 A.W. Arbo et al. The dataset experiment shows that including CARLA data as a compo- nent when training the perception models increases the total performance. As the model’s goal is to make good predictions in both real and simulated en- vironments, combining data from both seems to be a reasonable approach. CARLA+Aug achieves great results when evaluating on CARLA data, however the performance decreased drastically when predicting in real-world environ- ments. Models trained on real-world data tends to generalize better to unseen simulated environments than the other way around. Incorporating some CARLA data into the real-world data in addition to augmenting the images yields the best results overall. Experiment 1-3: Multi-task perception. Inspired by [15] we introduced a depth estimation decoder to the MobileNet-U-Net model. The model was trained with ground truth data generated by the Monodepth2 network using images from the Mapillary dataset, while depth maps for the CARLA data was included in the generated CARLA dataset. Segmentation Depth Training dataset Mean IoU Weighted IoU δ < 1.25 δ < 1.252 δ < 1.253 Mapillary 0.458 (+0.03) 0.817 0.320 0.572 0.684 Mapillary+CARLA 0.520 (+0.05) 0.854 0.295 0.542 0.679 CARLA 0.717 (+0.15) 0.960 0.775 0.806 0.816 Table 3: Results after adding a depth estimation decoder to the Mobilenet-U-Net model. Each model was trained on the same dataset as in Experiment 2. Mean IoU additionally presents a difference in parantheses: the difference between these models’ mean IoU and their counterparts’ from Experiment 1. Depth is estimated using accu- racy within threshold, where the set threshold is presented in the column title. A high value is best for all metrics in the table. Including a depth estimation decoder increases the segmentation performance for each model. The mean increase in IoU on the CARLA test set is 8%, which conforms with the results reported by [24], who reported a 4.17% increase in performance when training semantic segmentation with depth estimation. An increase in overall scene understanding can also be expected as depth is intro- duced to the model, however this has to be verified as part of the overall driving policy experiments. 4.2 Experiment 2: Driving Model This experiment aims to assess the overall performance of the two-part (percep- tion and driving policy) architecture. We run real-time evaluations on variants Domain-Independent Perception in Autonomous Driving 11 of our proposed architecture, including a baseline network where the complete network is trained at once. The evaluation is conducted with a custom sce- nario runner for CARLA, originally introduced by [14], and extended for our experiments. The real-time nature of this experiment makes it different from the previous experiments: The models’ steering and speed outputs affect the camera input in subsequent simulation steps, and each prediction is therefore dependent on the ones before. The scenario runner. The scenario runner makes each model drive through a predefined set of routes, each of which is defined by a set of waypoints. The model navigates each route using HLCs provided automatically when passing each waypoint. Each attempt at a route ends either when the vehicle completes the route, or when the vehicle enters any of the following erroneous states: stuck on an obstacle, leaving its correct lane and not returning within five seconds, or ignoring a HLC. The models are then compared on their mean route completion rate. Environments and Routes. The models were tested in two environments, Town02 and Town07. Town02 is similar, but not identical to the one in the driving policy’s training data, which is Town01. Town07 is quite different, and is rural with narrow roads (some without any centre marking), fields, and barns. There are three routes in each environment, which the models will try to com- plete in six different weather conditions. Three of the weather conditions have already been observed in the training data, while the three remaining are un- known to the policy. The training data only contain samples from day-time weathers, but two of the unknown weathers are at midnight. Results. Table 4 summarizes the driving performance of the different models. The model trained only on driving data and without a frozen perception model, RGB, was the best-performing model on Town02, but it struggles with Town07. The model names starting with SD indicates that they use the segmen- tation and depth perception model (henceforth SD). SD-CARLA, which uses SD trained only on perception data from CARLA, outperforms all other models when ranked by Mean Completion Rate (MCR) over both towns. To demonstrate its performance, we made a video (https://youtu.be/HL5LStDe7wY) showing some of its good performing moments. SD-Mapillary uses SD as well, but only had perception training data from Mapillary. While not performing as good as SD-CARLA, it still has impressive results. Its perception model has not seen any CARLA data, but is still able to predict segmentation and depth good enough for the driving model to beat even the RGB model. SD-Combined used percep- tion data from both Mapillary and CARLA, and performs a little bit worse than SD-Mapillary. The model names starting with S indicates that they use the segmentation- only perception model (henceforth S). S-CARLA is the S-counterpart of SD- CARLA, and it performs very well in Town02. In Town07 however, it struggles 12 A.W. Arbo et al. Seen weather Unseen weather Model Clear (D) Rain (D) Wet (S) Clear (N) Rain (N) Fog (S) Mean RGB 100.00 % 28.72 % 36.05 % 100.00 % 11.22 % 100.00 % 62.67 % SD-CARLA 100.00 % 43.30 % 55.36 % 100.00 % 23.46 % 44.96 % 61.18 % S-CARLA 93.83 % 7.23 % 43.51 % 100.00 % 24.61 % 66.66 % 55.97 % SD-Mapillary 88.51 % 42.16 % 67.22 % 100.00 % 11.59 % 24.91 % 55.73 % SD-Combined 93.53 % 9.92 % 47.42 % 100.00 % 9.92 % 46.53 % 51.22 % S-Combined 90.71 % 44.02 % 23.12 % 72.12 % 2.72 % 7.23 % 39.98 % S-Mapillary 72.12 % 43.30 % 40.85 % 39.34 % 2.72 % 2.72 % 33.51 % (a) Results for Town02. Seen weather Unseen weather Model Clear (D) Rain (D) Wet (S) Clear (N) Rain (N) Fog (S) Mean SD-CARLA 84.95 % 61.14 % 84.95 % 60.81 % 88.88 % 17.19 % 66.32 % SD-Mapillary 77.28 % 77.28 % 55.54 % 51.61 % 33.33 % 14.84 % 51.65 % SD-Combined 50.52 % 61.14 % 57.39 % 38.60 % 66.67 % 33.33 % 51.27 % S-Combined 77.83 % 55.10 % 55.10 % 17.32 % 50.65 % 33.33 % 48.22 % RGB 44.49 % 44.49 % 44.49 % 44.49 % 49.75 % 44.49 % 45.37 % S-CARLA 55.10 % 55.10 % 57.39 % 17.32 % 17.32 % 61.87 % 44.02 % S-Mapillary 51.61 % 50.52 % 44.49 % 55.10 % 44.49 % 0.00 % 41.04 % (b) Results for Town07. Table 4: Mean completion rate in (a) Town02 and (b) Town07, in six weather condi- tions. Day, Sunset and Night is shortened to D, S, N respectively. The individual cells are colored on a scale where green is the best, and red is the worst. Note that no cars or pedestrians were included in the traffic during these experiments. Domain-Independent Perception in Autonomous Driving 13 with night-time weather. S-Mapillary is the S-counterpart of SD-Mapillary, and it has the lowest MCR in both towns. In any run with Fog (S), it fails almost imme- diately. S-Combined uses combined perception data, the same as SD-Combined. It is performing a bit better than S-Mapillary in Town02, and is the fourth best in Town07. 5 Discussions 5.1 Results in comparison to related work [26] uses ground-truth semantic segmentation data generated from CARLA, not predicted as we do, and combine segmentation with both ground-truth depth maps and depth estimated by a separate network. Their results aligns with our results; using semantic segmentation data beats just using raw images, and combining both segmentation and depth performs the best. With a combination of ground truth segmentation and estimated depth, their policy is still able to beat the raw image-based policy. Our models estimate both segmentation and depth, and is still able to perform good in comparison to our baseline RGB- model. [20] use predicted binary segmentation (road/not road) as driving input, and our work extends this with predicted depth, giving additional performance benefits. [14] achieved higher completion rates even with traffic, but focused more on the impact of larger datasets and encoding temporal information in the model, while this paper focused mainly on generalizability. The driving model by [15] did not include the perception model’s decoding layers in its architecture, which seems to be an overall more efficient approach. Because the U-Net architecture used in our paper had connections between each encoder-decoder layer, information could have been lost by not including the de- coding layers. In future work, a model without connections between the encoder- decoder layers could be explored to take advantage of [15]’s approach. 5.2 Driving models We find that models with a learned understanding of both the semantics and/or geometry of the scene are able to navigate never-before-seen environments and weather. Our real-time experiment shows that these driving models often per- form better than learning from raw image inputs directly, with models utilizing both semantics and geometry performing best overall. Variance. It is important to note that we observed a high variance when train- ing and evaluating our models. Two models trained from the exact same setup could perform significantly different, despite having the exact same training data. We suspect that this is the same variance problem as [7] experienced. The vari- ance was handled by training and testing the models several times to make sure 14 A.W. Arbo et al. the results were representative. Still, conclusions based on the results in Exper- iment 2 must be drawn carefully. A more robust approach could be to train multiple models with the same parameters, and averaging their results. 6 Conclusion Splitting end-to-end models for autonomous vehicles into separate models for perception and driving policy is shown to give good results in simulated en- vironments. Perception models trained from public datasets such as Mapillary Vistas can be used to reduce the amount of driving data needed when training an end-to-end driving policy network. This approach opens up for training the driving policy in a simulated environment, while still getting good performance in real-world environments. Future work should explore how these results transfers into the real world. Evaluating the performance of a model trained solely in simulation directly in a real-world environment will be an important next step as a means of testing the validity of these results. References 1. Alhashim, I., Wonka, P.: High quality monocular depth estimation via transfer learning. arXiv:1812.11941 [cs] (Mar 2019), arXiv: 1812.11941 2. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv:1511.00561 [cs] (Oct 2016), arXiv: 1511.00561 3. Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., Zieba, K.: End to End Learning for Self-Driving Cars (2016) 4. Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al.: Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In: 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA). p. 4243–4250 (May 2018). https://doi.org/10.1109/ICRA.2018.8460875 5. Cao, Y., Zhao, T., Xian, K., Shen, C., Cao, Z., Xu, S.: Monocular depth estima- tion with augmented ordinal depth relationships. arXiv:1806.00585 [cs] (Jul 2019), arXiv: 1806.00585 6. Codevilla, F., Müller, M., López, A., Koltun, V., Dosovitskiy, A.: End-to-end Driv- ing via Conditional Imitation Learning. arXiv:1710.02410 [cs] (Oct 2017), arXiv: 1710.02410 7. Codevilla, F., Santana, E., López, A.M., Gaidon, A.: Exploring the Limitations of Behavior Cloning for Autonomous Driving. arXiv:1904.08980 [cs] (Apr 2019), arXiv: 1904.08980 8. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. arXiv:1604.01685 [cs] (Apr 2016), arXiv: 1604.01685 9. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning. pp. 1–16 (2017) Domain-Independent Perception in Autonomous Driving 15 10. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR) (2013) 11. Godard, C., Mac Aodha, O., Firman, M., Brostow, G.: Digging into self-supervised monocular depth estimation. arXiv:1806.01260 [cs, stat] (Aug 2019), arXiv: 1806.01260 12. Gualtieri, M., Pas, A.t., Saenko, K., Platt, R.: High precision grasp pose detection in dense clutter. arXiv:1603.01564 [cs] (Jun 2017), arXiv: 1603.01564 13. Gupta, D.: Image segmentation keras : Implementation of segnet, fcn, unet, pspnet and other models in keras. (2020), https://github.com/divamgupta/ image-segmentation-keras 14. Haavaldsen, H., Aasboe, M., Lindseth, F.: Autonomous Vehicle Control: End-to- end Learning in Simulated Urban Environments. arXiv:1905.06712 [cs] (May 2019), arXiv: 1905.06712 15. Hawke, J., Shen, R., Gurau, C., Sharma, S., Reda, D., Nikolov, N., Mazur, P., Mick- lethwaite, S., Griffiths, N., Shah, A., Kendall, A.: Urban Driving with Conditional Imitation Learning. arXiv:1912.00177 [cs] (Dec 2019), arXiv: 1912.00177 16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv:1512.03385 [cs] (Dec 2015), arXiv: 1512.03385 17. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., An- dreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 [cs] (Apr 2017), arXiv: 1704.04861 18. Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S.N., Rosaen, K., Vasudevan, R.: Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? arXiv:1610.01983 [cs] (Feb 2017), arXiv: 1610.01983 19. Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. arXiv:1705.07115 [cs] (Apr 2018), arXiv: 1705.07115 20. Müller, M., Dosovitskiy, A., Ghanem, B., Koltun, V.: Driving Policy Transfer via Modularity and Abstraction. arXiv:1804.09364 [cs] (Dec 2018), arXiv: 1804.09364 21. Neuhold, G., Ollmann, T., Rota Bulò, S., Kontschieder, P.: The mapillary vistas dataset for semantic understanding of street scenes. In: International Conference on Computer Vision (ICCV) (2017), https://www.mapillary.com/dataset/vistas 22. Pomerleau, D.A.: Advances in Neural Information Processing Systems 1, p. 305–313. Morgan Kaufmann Publishers Inc. (1989), http://dl.acm.org/ citation.cfm?id=89851.89891 23. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. arXiv:1505.04597 [cs] (May 2015), arXiv: 1505.04597 24. Standley, T., Zamir, A.R., Chen, D., Guibas, L., Malik, J., Savarese, S.: Which tasks should be learned together in multi-task learning? arXiv:1905.07553 [cs] (May 2019) 25. Viereck, U., Pas, A.t., Saenko, K., Platt, R.: Learning a visuomotor controller for real world robotic grasping using simulated depth images. arXiv:1706.04652 [cs] (Nov 2017), arXiv: 1706.04652 26. Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., López, A.M.: Multimodal End- to-End Autonomous Driving. arXiv:1906.03199 [cs] (Jun 2019), arXiv: 1906.03199 27. Yurtsever, E., Lambert, J., Carballo, A., Takeda, K.: A Survey of Autonomous Driving: Common Practices and Emerging Technologies. arXiv:1906.05113 [cs, eess] (Jun 2019), arXiv: 1906.05113