Deep convolutional Q-learning for traffic lights optimization in Smart Cities Riccardo Cappi1 , Sebastiano Monti1 and Davide Tosi2 1 University of Padova, Padova - Italy 2 Università degli studi dell’Insubria, Varese - Italy Abstract Autonomous traffic control is an important and active field of research that could potentially lead to remarkable improvements in congestion management and consequent delay and air pollution reductions. In this paper, we propose a deep reinforcement learning model to achieve autonomous traffic lights control at an intersection in a simulated environment. The model consists of a Convolutional Neural Network (CNN) that takes as input an image-like representation of the traffic state and is trained, using the Deep Q-Learning algorithm (DQL), to maximize a reward function based both on the decrease in queue length and maximum waiting times. We show that this approach reduces average waiting time and average queue length when compared to several baselines, such as a multi-layer perceptron architecture with a simpler state space representation and four non-parametric models, which implement the most waiting first heuristic, the longest queue first heuristic, an actuated traffic control scheme, and a simple static configuration of the traffic lights, respectively. The designed approach suggests its applicability in future smart cities for real traffic light control systems. Keywords Reinforcement Learning, Deep Q-learning, Traffic lights, Convolutional Neural Networks, Smart Cities 1. Introduction The advancement of smart technologies during the past 10 years, such as IoT devices, big data analytics, and artificial intelligence methods, has led to the emergence of Smart Cities. One of the key components of the smart urban environment is the optimization of urban vehicle transportation, directly impacting traffic congestion, costs, and emissions [1]. Two types of solutions are possible to address this challenge. The least efficient one, in terms of costs and durability, consists in the expansion of road infrastructures, while the most functional one involves increasing the efficiency of already existing infrastructures, such as traffic light signals at intersections [2]. The latter can be implemented through several algorithms, such as static traffic light phases or vehicle-actuated signal control. However, the most promising techniques for adaptive signal control seem to be based on Reinforcement Learning (RL) [3]. This paper aims at implementing a RL-based agent able to dynamically control the traffic light phases of an intersection in order to minimize jam lengths and vehicles’ waiting times. In particular, we implemented a Convolutional Neural Network (CNN), trained using the Deep Q-Learning (DQL) algorithm, which takes as input an image-like representation of the traffic state. We employed a state space definition that combines discrete traffic state encoding (DTSE) [2] with vehicles’ waiting times in order to consider both space and time information. We also defined a reward function according to the best-performing approaches proposed in literature, which involves both the variation in queue length and waiting times. We evaluated the performance of our model by comparing it with that of different baselines, such as a multi-layer perceptron architecture with a simpler state space representation and four heuristic-based models. We show that our approach performs better than the baselines in reducing the average queue length and the average waiting time at the considered intersection. ATT’24: Workshop Agents in Traffic and Transportation, October 19, 2024, Santiago de Compostela, Spain $ riccardo.cappi@studenti.unipd.it (R. Cappi); sebastiano.monti@studenti.unipd.it (S. Monti); davide.tosi@uninsubria.it (D. Tosi) € https://github.com/riccardocappi/Deep_Reinforcement_Learning_For_Traffic_Lights (R. Cappi)  0000-0002-3718-5892 (R. Cappi) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings The next sections are organized as follows: Section 2 summarizes the most common algorithms and methodologies present in literature in the field of adaptive traffic lights control. Section 3 briefly describes the reinforcement learning paradigm. Section 4 defines the components of the operating environment in which the agent works, such as the performance measures and the employed simulation software. Section 5 provides details regarding the state space, action space and reward function, as well as describing the learning algorithm and network architecture. Section 6 details the experimental setup and the obtained results, while Section 7 summarizes the conducted research. 2. Related works A lot of research has been done using reinforcement learning to build adaptive traffic signal control systems. These works mainly differ in the state representation of the environment, the action space of the agent and the reward function. Authors in [4][5] defined the state representation on the basis of queue length of different incoming roads, while in [6] the traffic state is estimated by considering both queue length and the maximum time a vehicle has waited on each lane at the intersection. However, authors in [2] pointed out that these abstract representations of the traffic state may omit relevant information and lead to suboptimal solutions. For this reason, other works employed an image-like representation by defining a Boolean-valued matrix whose cells can contain a value of one, indicating the presence of a vehicle, or zero, indicating its absence [7]. In [2][8], this matrix is further combined with another that indicates vehicles’ speed at the intersection. In this paper, instead, we aim at developing a model able to automatically learn high-level state representations without providing as input too many handcrafted features. To this purpose, we implemented a convolutional neural network that takes as input an image-like representation of the traffic state, exploiting the idea mentioned above. However, we propose a state definition that takes into consideration both the position and the waiting times of vehicles, and additionally uses a stack of consecutive simulation frames to make the model able to implicitly estimate vehicles’ velocity and travel direction, following the idea proposed in [9]. An important aspect of reinforcement learning for traffic lights control is how the action space is defined. Previous works proposed two different possibilities: (1) authors in [8] proposed a system in which all the phases cyclically change in a fixed sequence to guide vehicles through the intersection. In that system, the agent’s action is to select the phase duration in the next cycle. (2) On the other hand, most of the previous research defined the action space as the set of possible signal phase configurations (i.e., all the allowed green/red light configurations at the intersection) [7][9][2]. In this scenario, the agent’s action consists in selecting which lanes get a green light by choosing one of the allowed green/red light settings. Since the agent does not optimize the duration of each phase, green/red light timings can only be a multiple of a fixed-length interval. We chose to use the second action space definition, as it seems to be the most popular. Another key component is the reward function. A lot of reward definitions have been proposed in literature, such as change in cumulative vehicle delay [10][9] and change in number of queued vehicles [5]. However, authors in [6] suggest to define a reward function that is based both on the decrease in queue length and on the decrease in vehicles’ waiting times. This approach is also proposed in [11], where the results show that if the reward is exclusively based on queue length metrics, the model could leave some cars wait for an indefinite period of time. Therefore, in order to avoid situations of this kind, we decided to design our reward function following the latter approach. 3. Background In a reinforcement learning setting, an agent interacts with the environment to get rewards from its actions. Usually, a reinforcement learning model faces an unknown Markov decision process. It consists of the set of all the states 𝑆, the action set 𝐴, the transition function 𝛿, and the reward function 𝑅. At each discrete time 𝑡: • the agent observes state 𝑠𝑡 ∈ 𝑆; • it chooses action 𝑎𝑡 ∈ 𝐴 (among the possible actions in state 𝑠𝑡 ) and executes it; • it receives an immediate reward 𝑟𝑡 = 𝑅(𝑠𝑡 , 𝑎𝑡 ), that can be positive, negative or neutral; • the state changes to 𝑠𝑡+1 = 𝛿(𝑠𝑡 , 𝑎𝑡 ). Assuming that 𝑟𝑡 and 𝑠𝑡+1 only depend on current state and action, the agent’s goal is to learn an action policy 𝜋 : 𝑆 → 𝐴 that maximizes the expected sum of (discounted) rewards obtained if policy 𝜋 is followed. For each possible policy 𝜋 the agent might adopt, we can define an evaluation function over states: ∞ ∑︁ 𝑉 𝜋 (𝑠) = 𝛾 𝑖 𝑟𝑡+𝑖 (1) 𝑖=0 where 𝑟𝑡 , 𝑟𝑡+1 , ... are generated executing policy 𝜋 starting at state 𝑠. Then, the choice of the best actions to play becomes an optimization problem. Indeed, it comes down to finding the optimal policy 𝜋 * that maximizes (1) for all states 𝑠: 𝜋 * = 𝑎𝑟𝑔𝑚𝑎𝑥𝜋 𝑉 𝜋 (𝑠), (∀𝑠). (2) In the Q-learning framework, a numeric value 𝑄(𝑠, 𝑎) ∈ R, called Q-value, is associated to each state-action pair. The value of 𝑄 is the reward received immediately upon executing action 𝑎 from state 𝑠, plus the value (discounted by 𝛾) of following the optimal policy thereafter: * 𝑄(𝑠, 𝑎) = 𝑅(𝑠, 𝑎) + 𝛾𝑉 𝜋 (𝛿(𝑠, 𝑎)) where 𝛿(𝑠, 𝑎) denotes the state resulting from applying action 𝑎 to state 𝑠. Then, we can reformulate (2) as: 𝜋 * (𝑠) = 𝑎𝑟𝑔𝑚𝑎𝑥𝑎 𝑄(𝑠, 𝑎). The Q-values are estimated in the Q-learning algorithm by iterative Bellman updates: 𝑄𝑡 (𝑠, 𝑎) = 𝑄𝑡−1 (𝑠, 𝑎) + 𝛼(𝑟 + 𝛾𝑚𝑎𝑥𝑎′ 𝑄𝑡−1 (𝑠′ , 𝑎′ ) − 𝑄𝑡−1 (𝑠, 𝑎)). * In this way, if the agent learns the 𝑄 function instead of 𝑉 𝜋 , it will be able to select optimal actions even if it has no knowledge of 𝑅 and 𝛿. 4. Operating environment In this section, we define the operating environment in which the agent works. Simulation environment: since it is difficult to retrieve real traffic data and perform real-world experimentation, we relied on SUMO [12], an open source traffic simulator that makes it possible to model real-world traffic behavior. This software, through an API called TraCI, provides complete control over the simulation environment elements, such as vehicles’ speed and position, traffic flow’s intensity on each lane, traffic light phases, the shape of the intersection, etc. Performance measures: the performance of the agent is assessed with respect to two common traffic metrics: queue length and vehicles’ waiting times. The goal is to find a model able to dynamically control the traffic lights of an intersection in order to minimize these two metrics. Although dynamic traffic light control is an extremely complex task in the real world, SUMO allows you to operate in a more controlled environment. Specifically, the agent works in a fully-observable environment, since the software gives access to its complete state at each point in time. For this paper, we also defined a deterministic environment by setting a non-stochastic traffic flow generation. This makes the analysis simpler, but it is also one of the biggest limitations of this work. Clearly, the environment is also sequential and single agent. 5. Methods In order to build a reinforcement learning model for traffic lights control, we need to define the traffic state representation, the action space and the reward function. 5.1. State space We propose a state representation that takes into consideration both vehicles’ positions and waiting times. The idea is to map each lane approaching the intersection into a Boolean-valued vector, where each cell can contain a 1, indicating the presence of a vehicle at that position, or a 0, indicating its absence. Each cell of the vector corresponds to 1 meter of the lane. The matrix of vehicles’ positions is then obtained by stacking all the lane vectors. Given an intersection with 𝑙 lanes, where the longest lane is 𝑚 meters, this intermediate state representation 𝑠′ consists of a (𝑙 × 𝑚) matrix. Note that a zero-padding is added to lane vectors with a length less than the maximum length lane in order to have all equally-sized vectors. Then, the 𝑠′ representation is enriched by using a stack of consecutive simulation frames to make the model able to implicitly estimate vehicles’ velocity and travel direction. In particular, 𝑠′ is computed for the last 𝑝 (𝑝 = 2 in our setting) simulation steps, yielding a new (𝑝 × 𝑙 × 𝑚) matrix, denoted as 𝑠′′ . The 𝑠′′ representation built so far consists of a Boolean-valued matrix that contains the information about vehicles’ positions of the last 𝑝 simulation steps. However, it does not take into consideration the waiting times. This information is embodied in the representation by computing another state matrix, whose cells contain the normalized values of the vehicles’ waiting times of the last simulation step. Then, the final state representation 𝑠 is a (𝑝 + 1 × 𝑙 × 𝑚) matrix. 5.2. Action space To handle traffic at the intersection, the agent selects which lanes get a green light according to a set of three possible green/red light configurations. On each of the three incoming roads there is a traffic light that manages the traffic on the corresponding lanes. The combination of the individual phases of these traffic lights forms the set of the possible green/red light configurations. In Figure 1, all the three Figure 1: Possible phase configurations that can occur at the intersection possible signal phases that can occur at the considered intersection are shown. Green and red lines represent the routes that vehicles can travel during the simulation. Vehicles on green paths are allowed to pass, while vehicles on red paths must stop. 5.3. Reward function The proposed definition of the reward function takes into account both the variation in queue length and waiting times. In particular, the reward 𝑟𝑡 is given by the following formula: 𝑟𝑡 = (𝐽𝑡 − 𝐽𝑡+1 ) − 𝛼𝑊𝑡+1 where 𝐽𝑡 represents the sum of the jam lengths (in meters) observed over the lanes at time 𝑡, and 𝑊𝑡+1 represents the sum of the maximum waiting times (in seconds) observed over the lanes at time 𝑡 + 1. 𝛼 is a hyper-parameter that determines how much to penalize the agent for letting vehicles wait too much (in our setting 𝛼 = 0.4). The agent receives a positive reward if the last action performed, 𝑎𝑡 , leads to a state 𝑠𝑡+1 with shorter total queue length and/or low maximum waiting times. 5.4. Network architecture The proposed architecture is a convolutional neural network that takes as input the state matrix mentioned in Section 5.1 and returns as output an approximation of the optimal Q-values. The model is composed of two convolutional layers and two fully connected layers at the end. In particular, the first convolutional layer consists of 16 (2 × 10)-filters with stride (2 × 1) followed by a LeakyReLU activation function. The second layer has 32 (1 × 4)-filters with stride (1 × 2) followed by a LeakyReLU activation function and a max pooling layer of size (1 × 2). The first fully-connected layer has 256 nodes followed by a LeakyReLU activation function, while the output layer has 3 linear output neurons (one for each possible green/red light configuration). In Figure 2, a summary of the CNN architecture is shown. We designed the convolutional kernels so that, ideally, they compute high-level representations of each road separately. Then, the joint information among the different roads is merged by the network in the last two fully connected layers. Flatten Dense Output Max Pooling Conv 2 Conv 1 Input Figure 2: CNN architecture summary 5.5. Learning algorithm The proposed model was trained using the Deep Q-Learning (DQL) algorithm, shown in Algorithm 1, which consists in combining Q-Learning with Deep Neural Networks (DNN). The employed hyper- parameters are shown in Table 1, which are typically found in literature. 6. Experiments 6.1. Simulation setup The considered intersection (Figure 1) is composed of three incoming roads, each with two lanes. In order to simulate real-life scenarios, the intersection was designed similarly to a real one located in Como (IT) at the following coordinates: (45.802155, 9.084961). The two main roads’ lengths are Algorithm 1 Deep Q-Learning with Experience Replay 1: procedure DQL for traffic lights control 2: Initialize replay memory 𝐷 to capacity 𝐿 3: Initialize policy network 𝑄 with random weights 𝜃 4: Initialize target network 𝑄 ^ with random weights 𝜃− = 𝜃 5: Create simulation environment 𝑒𝑛𝑣 6: 𝑒𝑝𝑜𝑐ℎ ← 0 7: 𝑒𝑝𝑖𝑠𝑜𝑑𝑒 ← 0 8: while 𝑒𝑝𝑜𝑐ℎ < 𝑁 do 9: 𝑠1 ← 𝑒𝑛𝑣.reset() 10: 𝑒𝑝𝑖𝑠𝑜𝑑𝑒 ← 𝑒𝑝𝑖𝑠𝑜𝑑𝑒 + 1 11: for 𝑡 = 1, 𝑇 do 12: Select action 𝑎𝑡 sampling from 𝑠𝑜𝑓 𝑡𝑚𝑎𝑥(𝑄(𝑠𝑡 ; 𝜃)) 13: 𝑠𝑡+1 , 𝑟𝑡 ← 𝑒𝑛𝑣.step(𝑎𝑡 ) 14: Store transition ⟨𝑠𝑡 , 𝑎𝑡 , 𝑟𝑡 , 𝑠𝑡+1 ⟩ in 𝐷 15: Sample a mini-batch of transitions ⟨𝑠𝑗 , 𝑎𝑗 , 𝑟𝑗 , 𝑠𝑗+1 ⟩ uniformly from 𝐷 16: if 𝑠𝑗+1 is terminal then 17: 𝑦𝑗 ← 𝑟𝑗 18: else 19: 𝑦𝑗 ← 𝑟𝑗 + 𝛾 max𝑎′ 𝑄 ^ (𝑠𝑗+1 , 𝑎′ ; 𝜃− ) 20: 𝑙𝑜𝑠𝑠 = 𝑠𝑚𝑜𝑜𝑡ℎ_𝐿1_𝑙𝑜𝑠𝑠(𝑦𝑗 , 𝑄(𝑠𝑗 , 𝑎𝑗 ; 𝜃)) 21: Optimize 𝜃, using ADAM, according to 𝑙𝑜𝑠𝑠 22: 𝜃− ← 𝜏 𝜃 + (1 − 𝜏 ) 𝜃− 23: if 𝑒𝑝𝑖𝑠𝑜𝑑𝑒 mod 5 = 0 then 24: 𝑒𝑝𝑜𝑐ℎ ← 𝑒𝑝𝑜𝑐ℎ + 1 25: Evaluate 𝑄 Table 1 Agent’s hyper-parameters Hyper-parameter Value Optimizer ADAM Replay memory size 5000 Learning rate 0.001 Mini-Batch size 32 Discount factor 𝛾 0.9 State matrix size 3 × 6 × 309 Epochs 45 𝜏 0.001 309𝑚 and 211𝑚 respectively, while the minor road’s length is 103𝑚. The max speed is 13.9𝑚/𝑠, which is equal to 50𝑘𝑚/ℎ, on each road. On each lane, vehicles can travel following different routes through the intersection. Due to the difficulty of finding a dataset of the traffic flows of Italian roads, we set the traffic flow rate to 450 vehicles per hour on each route. A scheme of the routes that vehicles can travel is shown in Figure 1. We can observe that the east incoming road has 4 different routes; therefore, the traffic on that road will be higher than on the others. The minimum green/red-light phase duration is fixed at 10 simulation steps (10 seconds in the simulation environment), while the yellow-light phase duration between two neighboring phases is fixed at 5 seconds. These two fixed lengths determine how many simulation steps SUMO can run before letting the model take a new action. With this configuration, the green-light phase is guaranteed to be of at least 10 seconds. For simplicity, we chose to generate only one vehicle’s type. In particular, each vehicle’s length is 5 meters. After 500 simulation steps, the system stops generating vehicles and the simulation ends. The proposed model was trained for 45 epochs, where each epoch is composed of 5 complete SUMO simulations. 6.2. Results As we said before, the proposed model was assessed with respect to two common traffic metrics: queue length and vehicles’ waiting times. We compared the performance of the proposed model with that of the following baselines: • A Multi-Layer Perceptron (MLP) network with one fully-connected hidden layer of 80 nodes, followed by a ReLU activation function, and 3 linear output neurons. The input of the MLP consists of a vector containing the information about the current phase, the queue length (in meters) on each lane, and the maximum time (in seconds) a vehicle has waited on each lane at the intersection, following the approach proposed in [6]. The MLP was trained with the same hyper-parameters and optimization method used for the CNN. • Two traffic control systems provided by default by SUMO: (1) the first one is a simple Static configuration of traffic light signals, in which all the phases cyclically change in a fixed sequence and each green/red-light phase has a fixed duration of 25 seconds, while the yellow-light duration is still 5 seconds. (2) The second system is the default implementation of the gap-based Actuated traffic control scheme, which dynamically adjusts traffic light phases’ durations whenever a continuous stream of traffic is detected. • Two models that implement the most waiting first (MWF) heuristic and longest queue first (LQF) heuristic. The first model sets a green light to lanes in which vehicles waited the most, up to the current simulation step. The second model, instead, sets a green light to lanes in which the longest queues were observed. For both models, the green/red-light duration and the yellow-light duration are the same as the CNN model. Table 2 Performance comparison of the analyzed models. CNN and MLP’s values are obtained by testing the models that got the highest average reward during the training phase. Model Max queue length Max waiting time Avg. queue length Avg. waiting time [𝑚] [𝑠] [𝑚] [𝑠] CNN 87.54 70 15.78 13.49 MLP 124.33 159 20.71 19.41 MWF 181.21 157 38.76 36.67 LQF 140.01 241 36.65 41.17 Static 199.43 214 31.15 30.94 Actuated 178.21 117 27.67 26.14 Table 2 shows the performance of the tested models. It is clear that the proposed agent performs better than every baseline, providing less average waiting time1 and less average queue length. We can also see that the non-parametric methods such as MWF, LQF, Actuated and Static heuristics perform dramatically worse than the RL-based agents. Therefore, we continue the analysis by exploring the differences between the two neural network models. In Figure 3, a comparison between the average rewards obtained by the CNN and the MLP on each epoch is shown (red line and blue line, respectively). The learning process seems to be more stable for the CNN-based agent, which performs better than the baseline. However, we can observe a rapid increase in the rewards obtained by the MLP agent at the end of the training. This suggests that, even 1 the average waiting time at the intersection is computed by averaging the maximum waiting times observed on each lane during the simulation Figure 3: Average rewards obtained by the CNN and the MLP over the epochs. if the CNN model provides better results in this experiment, the MLP does not perform dramatically worse. The same result can be deduced by looking at the average queue lengths and average waiting times obtained by the two architectures over the epochs, shown in Figure 4. For this reason, in order to Figure 4: Average queue length (meters) and average waiting time (seconds) obtained by CNN and MLP on each epoch assess whether the CNN-based agent concretely brings significant improvements with respect to the MLP-based one, we compared both models by training them under different traffic conditions. Figure 5 shows the box plots of the average rewards obtained by training both models in 4 different simulation setups, featuring increasing traffic intensities. Each setup is equivalent to the one presented in Section 6.1, with 350, 450, 550 and 700 vehicles per hour, respectively. The results show that, under low traffic conditions, the two models perform very similarly. However, the CNN-based agent scales better than the baseline with increasing traffic intensity, showing that the proposed model is more robust and can deal with more complex scenarios. 7. Conclusions Smart cities and the planet urgently ask for environmental emissions to be reduced while improving the quality of life for citizens. To this end, Artificial Intelligence can provide researchers with instruments and tools to help this virtuous process. In this paper, a new CNN-based approach has been designed and tested to improve queue length and vehicle waiting times for traffic light control systems. The Figure 5: Average rewards obtained by training CNN and MLP agents considering different traffic flow rates proposed approach has been extensively experimented against five baseline models. The results show that CNN models perform better than baselines. This opens the possibility of testing our approach in real-life conditions and in future Smart Cities that will exploit intelligent traffic light control systems. References [1] D. Tosi, Cell phone big data to compute mobility scenarios for future smart cities, International Jour- nal of Data Science and Analytics 4 (2017) 265–284. URL: https://doi.org/10.1007/s41060-017-0061-2. doi:10.1007/s41060-017-0061-2. [2] W. Genders, S. Razavi, Using a deep reinforcement learning agent for traffic signal control, arXiv preprint arXiv:1611.01142 (2016). [3] R. Chen, F. Fang, N. Sadeh, The real deal: A review of challenges and opportunities in mov- ing reinforcement learning-based traffic signal control systems towards reality, arXiv preprint arXiv:2206.11996 (2022). [4] Y. K. Chin, L. K. Lee, N. Bolong, S. S. Yang, K. T. K. Teo, Exploring q-learning optimization in traffic signal timing plan management, in: 2011 third international conference on computational intelligence, communication systems and networks, IEEE, 2011, pp. 269–274. [5] N. Maiti, B. R. Chilukuri, Traffic signal control for an isolated intersection using reinforcement learning, in: 2021 International Conference on COMmunication Systems & NETworkS (COM- SNETS), IEEE, 2021, pp. 629–633. [6] M. B. Natafgi, M. Osman, A. S. Haidar, L. Hamandi, Smart traffic light system using machine learning, in: 2018 IEEE International Multidisciplinary Conference on Engineering Technology (IMCET), IEEE, 2018, pp. 1–6. [7] E. Van der Pol, F. A. Oliehoek, Coordinated deep reinforcement learners for traffic light control, Proceedings of learning, inference and control of multi-agent systems (at NIPS 2016) 8 (2016) 21–38. [8] X. Liang, X. Du, G. Wang, Z. Han, Deep reinforcement learning for traffic light control in vehicular networks, arXiv preprint arXiv:1803.11115 (2018). [9] S. S. Mousavi, M. Schukat, E. Howley, Traffic light control using deep policy-gradient and value- function-based reinforcement learning, IET Intelligent Transport Systems 11 (2017) 417–423. [10] I. Arel, C. Liu, T. Urbanik, A. G. Kohls, Reinforcement learning-based multi-agent system for network traffic signal control, IET Intelligent Transport Systems 4 (2010) 128–135. [11] B. Koohy, S. Stein, E. Gerding, G. Manla, Reward function design in multi-agent reinforcement learning for traffic signal control (2022). [12] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, E. Wießner, Microscopic traffic simulation using sumo, in: The 21st IEEE International Conference on Intelligent Transportation Systems, IEEE, 2018. URL: https: //elib.dlr.de/124092/.