Autonomous trafic control is an important and active field of research that could potentially lead to remarkable improvements in congestion management and consequent delay and air pollution reductions. In this paper, we propose a deep reinforcement learning model to achieve autonomous trafic lights control at an intersection in a simulated environment. The model consists of a Convolutional Neural Network (CNN) that takes as input an image-like representation of the trafic state and is trained, using the Deep Q-Learning algorithm (DQL), to maximize a reward function based both on the decrease in queue length and maximum waiting times. We show that this approach reduces average waiting time and average queue length when compared to several baselines, such as a multi-layer perceptron architecture with a simpler state space representation and four non-parametric models, which implement the most waiting first heuristic, the longest queue first heuristic, an actuated trafic control scheme, and a simple static configuration of the trafic lights, respectively. The designed approach suggests its applicability in future smart cities for real trafic light control systems.
The advancement of smart technologies during the past 10 years, such as IoT devices, big data analytics,
and artificial intelligence methods, has led to the emergence of Smart Cities. One of the key components
of the smart urban environment is the optimization of urban vehicle transportation, directly impacting
trafic congestion, costs, and emissions [
The next sections are organized as follows: Section 2 summarizes the most common algorithms and methodologies present in literature in the field of adaptive trafic lights control. Section 3 briefly describes the reinforcement learning paradigm. Section 4 defines the components of the operating environment in which the agent works, such as the performance measures and the employed simulation software. Section 5 provides details regarding the state space, action space and reward function, as well as describing the learning algorithm and network architecture. Section 6 details the experimental setup and the obtained results, while Section 7 summarizes the conducted research.
A lot of research has been done using reinforcement learning to build adaptive trafic signal control
systems. These works mainly difer in the state representation of the environment, the action space of
the agent and the reward function. Authors in [
An important aspect of reinforcement learning for trafic lights control is how the action space is
defined. Previous works proposed two diferent possibilities: (1) authors in [
Another key component is the reward function. A lot of reward definitions have been proposed in
literature, such as change in cumulative vehicle delay [
In a reinforcement learning setting, an agent interacts with the environment to get rewards from its actions. Usually, a reinforcement learning model faces an unknown Markov decision process. It consists of the set of all the states , the action set , the transition function , and the reward function . At each discrete time : • the agent observes state ∈ ; • it chooses action ∈ (among the possible actions in state ) and executes it; • it receives an immediate reward = (, ), that can be positive, negative or neutral; • the state changes to +1 = (, ).
Assuming that and +1 only depend on current state and action, the agent’s goal is to learn an action policy : → that maximizes the expected sum of (discounted) rewards obtained if policy is followed. For each possible policy the agent might adopt, we can define an evaluation function over states: * = (), (∀). (2) In the Q-learning framework, a numeric value (, ) ∈ R, called Q-value, is associated to each state-action pair. The value of is the reward received immediately upon executing action from state , plus the value (discounted by ) of following the optimal policy thereafter:
(, ) = (, ) + * ( (, )) where (, ) denotes the state resulting from applying action to state . Then, we can reformulate (2) as:
* () = (, ).
The Q-values are estimated in the Q-learning algorithm by iterative Bellman updates: (, ) = − 1(, ) + ( + ′ − 1(′, ′) − − 1(, )).
In this way, if the agent learns the function instead of * , it will be able to select optimal actions even if it has no knowledge of and .
In this section, we define the operating environment in which the agent works.
Simulation environment: since it is dificult to retrieve real trafic data and perform real-world
experimentation, we relied on SUMO [
Performance measures: the performance of the agent is assessed with respect to two common trafic metrics: queue length and vehicles’ waiting times. The goal is to find a model able to dynamically control the trafic lights of an intersection in order to minimize these two metrics.
Although dynamic trafic light control is an extremely complex task in the real world, SUMO allows you to operate in a more controlled environment. Specifically, the agent works in a fully-observable environment, since the software gives access to its complete state at each point in time. For this paper, we also defined a deterministic environment by setting a non-stochastic trafic flow generation. This makes the analysis simpler, but it is also one of the biggest limitations of this work. Clearly, the environment is also sequential and single agent.
In order to build a reinforcement learning model for trafic lights control, we need to define the trafic state representation, the action space and the reward function.
We propose a state representation that takes into consideration both vehicles’ positions and waiting times. The idea is to map each lane approaching the intersection into a Boolean-valued vector, where each cell can contain a 1, indicating the presence of a vehicle at that position, or a 0, indicating its absence. Each cell of the vector corresponds to 1 meter of the lane. The matrix of vehicles’ positions is then obtained by stacking all the lane vectors. Given an intersection with lanes, where the longest lane is meters, this intermediate state representation ′ consists of a ( × ) matrix. Note that a zero-padding is added to lane vectors with a length less than the maximum length lane in order to have all equally-sized vectors.
Then, the ′ representation is enriched by using a stack of consecutive simulation frames to make the model able to implicitly estimate vehicles’ velocity and travel direction. In particular, ′ is computed for the last ( = 2 in our setting) simulation steps, yielding a new ( × × ) matrix, denoted as ′′.
The ′′ representation built so far consists of a Boolean-valued matrix that contains the information about vehicles’ positions of the last simulation steps. However, it does not take into consideration the waiting times. This information is embodied in the representation by computing another state matrix, whose cells contain the normalized values of the vehicles’ waiting times of the last simulation step. Then, the final state representation is a ( + 1 × × ) matrix.
To handle trafic at the intersection, the agent selects which lanes get a green light according to a set of three possible green/red light configurations. On each of the three incoming roads there is a trafic light that manages the trafic on the corresponding lanes. The combination of the individual phases of these trafic lights forms the set of the possible green/red light configurations. In Figure 1, all the three possible signal phases that can occur at the considered intersection are shown. Green and red lines represent the routes that vehicles can travel during the simulation. Vehicles on green paths are allowed to pass, while vehicles on red paths must stop.
The proposed definition of the reward function takes into account both the variation in queue length and waiting times. In particular, the reward is given by the following formula:
= ( − +1) − +1 where represents the sum of the jam lengths (in meters) observed over the lanes at time , and +1 represents the sum of the maximum waiting times (in seconds) observed over the lanes at time + 1. is a hyper-parameter that determines how much to penalize the agent for letting vehicles wait too much (in our setting = 0.4). The agent receives a positive reward if the last action performed, , leads to a state +1 with shorter total queue length and/or low maximum waiting times.
The proposed architecture is a convolutional neural network that takes as input the state matrix mentioned in Section 5.1 and returns as output an approximation of the optimal Q-values. The model is composed of two convolutional layers and two fully connected layers at the end. In particular, the ifrst convolutional layer consists of 16 (2 × 10)-filters with stride (2 × 1) followed by a LeakyReLU activation function. The second layer has 32 (1 × 4)-filters with stride (1 × 2) followed by a LeakyReLU activation function and a max pooling layer of size (1 × 2). The first fully-connected layer has 256 nodes followed by a LeakyReLU activation function, while the output layer has 3 linear output neurons (one for each possible green/red light configuration). In Figure 2, a summary of the CNN architecture is shown. We designed the convolutional kernels so that, ideally, they compute high-level representations of each road separately. Then, the joint information among the diferent roads is merged by the network in the last two fully connected layers.
Input
Conv 1
Conv 2
Max Pooling
Flatten
Dense
Output
The proposed model was trained using the Deep Q-Learning (DQL) algorithm, shown in Algorithm 1, which consists in combining Q-Learning with Deep Neural Networks (DNN). The employed hyperparameters are shown in Table 1, which are typically found in literature.
The considered intersection (Figure 1) is composed of three incoming roads, each with two lanes. In order to simulate real-life scenarios, the intersection was designed similarly to a real one located in Como (IT) at the following coordinates: (45.802155, 9.084961). The two main roads’ lengths are Algorithm 1 Deep Q-Learning with Experience Replay 1: procedure DQL for traffic lights control
Initialize replay memory to capacity Initialize policy network with random weights Initialize target network ^ with random weights − = Create simulation environment ℎ ← ← 0
0 while ℎ < do 1 ← ← for = 1, do .reset()
+ 1 Select action sampling from ((; )) +1, ←
.step() Store transition ⟨, , , +1⟩ in Sample a mini-batch of transitions ⟨ , , , +1⟩ uniformly from if +1 is terminal then else ←
← + max′ ^(+1, ′; − ) = ℎ_1_( , ( , ; ))
− ← + (1 − ) − if mod 5 = 0 then ℎ ←
ℎ + 1 309 and 211 respectively, while the minor road’s length is 103. The max speed is 13.9/, which is equal to 50/ℎ, on each road. On each lane, vehicles can travel following diferent routes through the intersection. Due to the dificulty of finding a dataset of the trafic flows of Italian roads, we set the trafic flow rate to 450 vehicles per hour on each route. A scheme of the routes that vehicles can travel is shown in Figure 1. We can observe that the east incoming road has 4 diferent routes; therefore, the trafic on that road will be higher than on the others. The minimum green/red-light phase duration is ifxed at 10 simulation steps ( 10 seconds in the simulation environment), while the yellow-light phase duration between two neighboring phases is fixed at 5 seconds. These two fixed lengths determine how many simulation steps SUMO can run before letting the model take a new action. With this configuration, the green-light phase is guaranteed to be of at least 10 seconds. For simplicity, we chose to generate only one vehicle’s type. In particular, each vehicle’s length is 5 meters. After 500 simulation steps, the system stops generating vehicles and the simulation ends. The proposed model was trained for 45 epochs, where each epoch is composed of 5 complete SUMO simulations.
As we said before, the proposed model was assessed with respect to two common trafic metrics: queue
length and vehicles’ waiting times. We compared the performance of the proposed model with that of
the following baselines:
• A Multi-Layer Perceptron (MLP) network with one fully-connected hidden layer of 80 nodes,
followed by a ReLU activation function, and 3 linear output neurons. The input of the MLP
consists of a vector containing the information about the current phase, the queue length (in
meters) on each lane, and the maximum time (in seconds) a vehicle has waited on each lane at
the intersection, following the approach proposed in [
In Figure 3, a comparison between the average rewards obtained by the CNN and the MLP on each epoch is shown (red line and blue line, respectively). The learning process seems to be more stable for the CNN-based agent, which performs better than the baseline. However, we can observe a rapid increase in the rewards obtained by the MLP agent at the end of the training. This suggests that, even 1the average waiting time at the intersection is computed by averaging the maximum waiting times observed on each lane during the simulation if the CNN model provides better results in this experiment, the MLP does not perform dramatically worse. The same result can be deduced by looking at the average queue lengths and average waiting times obtained by the two architectures over the epochs, shown in Figure 4. For this reason, in order to assess whether the CNN-based agent concretely brings significant improvements with respect to the MLP-based one, we compared both models by training them under diferent trafic conditions. Figure 5 shows the box plots of the average rewards obtained by training both models in 4 diferent simulation setups, featuring increasing trafic intensities. Each setup is equivalent to the one presented in Section 6.1, with 350, 450, 550 and 700 vehicles per hour, respectively. The results show that, under low trafic conditions, the two models perform very similarly. However, the CNN-based agent scales better than the baseline with increasing trafic intensity, showing that the proposed model is more robust and can deal with more complex scenarios.
Smart cities and the planet urgently ask for environmental emissions to be reduced while improving the quality of life for citizens. To this end, Artificial Intelligence can provide researchers with instruments and tools to help this virtuous process. In this paper, a new CNN-based approach has been designed and tested to improve queue length and vehicle waiting times for trafic light control systems. The proposed approach has been extensively experimented against five baseline models. The results show that CNN models perform better than baselines. This opens the possibility of testing our approach in real-life conditions and in future Smart Cities that will exploit intelligent trafic light control systems. J. Rummel, P. Wagner, E. Wießner, Microscopic trafic simulation using sumo, in: The 21st IEEE International Conference on Intelligent Transportation Systems, IEEE, 2018. URL: https: //elib.dlr.de/124092/.