The advancement of smart technologies during the past 10 years, such as IoT devices, big data analytics, and artificial intelligence methods, has led to the emergence of Smart Cities. One of the key components of the smart urban environment is the optimization of urban vehicle transportation, directly impacting traffic congestion, costs, and emissions [1]. Two types of solutions are possible to address this challenge. The least efficient one, in terms of costs and durability, consists in the expansion of road infrastructures, while the most functional one involves increasing the efficiency of already existing infrastructures, such as traffic light signals at intersections [2]. The latter can be implemented through several algorithms, such as static traffic light phases or vehicle-actuated signal control. However, the most promising techniques for adaptive signal control seem to be based on Reinforcement Learning (RL) [3]. This paper aims at implementing a RL-based agent able to dynamically control the traffic light phases of an intersection in order to minimize jam lengths and vehicles' waiting times. In particular, we implemented a Convolutional Neural Network (CNN), trained using the Deep Q-Learning (DQL) algorithm, which takes as input an image-like representation of the traffic state. We employed a state space definition that combines discrete traffic state encoding (DTSE) [2] with vehicles' waiting times in order to consider both space and time information. We also defined a reward function according to the best-performing approaches proposed in literature, which involves both the variation in queue length and waiting times. We evaluated the performance of our model by comparing it with that of different baselines, such as a multi-layer perceptron architecture with a simpler state space representation and four heuristic-based models. We show that our approach performs better than the baselines in reducing the average queue length and the average waiting time at the considered intersection.
The next sections are organized as follows: Section 2 summarizes the most common algorithms and methodologies present in literature in the field of adaptive traffic lights control. Section 3 briefly describes the reinforcement learning paradigm. Section 4 defines the components of the operating environment in which the agent works, such as the performance measures and the employed simulation software. Section 5 provides details regarding the state space, action space and reward function, as well as describing the learning algorithm and network architecture. Section 6 details the experimental setup and the obtained results, while Section 7 summarizes the conducted research.
A lot of research has been done using reinforcement learning to build adaptive traffic signal control systems. These works mainly differ in the state representation of the environment, the action space of the agent and the reward function. Authors in [4] [5] defined the state representation on the basis of queue length of different incoming roads, while in [6] the traffic state is estimated by considering both queue length and the maximum time a vehicle has waited on each lane at the intersection. However, authors in [2] pointed out that these abstract representations of the traffic state may omit relevant information and lead to suboptimal solutions. For this reason, other works employed an image-like representation by defining a Boolean-valued matrix whose cells can contain a value of one, indicating the presence of a vehicle, or zero, indicating its absence [7]. In [2][8], this matrix is further combined with another that indicates vehicles' speed at the intersection. In this paper, instead, we aim at developing a model able to automatically learn high-level state representations without providing as input too many handcrafted features. To this purpose, we implemented a convolutional neural network that takes as input an image-like representation of the traffic state, exploiting the idea mentioned above. However, we propose a state definition that takes into consideration both the position and the waiting times of vehicles, and additionally uses a stack of consecutive simulation frames to make the model able to implicitly estimate vehicles' velocity and travel direction, following the idea proposed in [9].
An important aspect of reinforcement learning for traffic lights control is how the action space is defined. Previous works proposed two different possibilities: (1) authors in [8] proposed a system in which all the phases cyclically change in a fixed sequence to guide vehicles through the intersection. In that system, the agent's action is to select the phase duration in the next cycle. (2) On the other hand, most of the previous research defined the action space as the set of possible signal phase configurations (i.e., all the allowed green/red light configurations at the intersection) [7][9] [2]. In this scenario, the agent's action consists in selecting which lanes get a green light by choosing one of the allowed green/red light settings. Since the agent does not optimize the duration of each phase, green/red light timings can only be a multiple of a fixed-length interval. We chose to use the second action space definition, as it seems to be the most popular.
Another key component is the reward function. A lot of reward definitions have been proposed in literature, such as change in cumulative vehicle delay [10] [9] and change in number of queued vehicles [5]. However, authors in [6] suggest to define a reward function that is based both on the decrease in queue length and on the decrease in vehicles' waiting times. This approach is also proposed in [11], where the results show that if the reward is exclusively based on queue length metrics, the model could leave some cars wait for an indefinite period of time. Therefore, in order to avoid situations of this kind, we decided to design our reward function following the latter approach.
In a reinforcement learning setting, an agent interacts with the environment to get rewards from its actions. Usually, a reinforcement learning model faces an unknown Markov decision process. It consists of the set of all the states π, the action set π΄, the transition function πΏ, and the reward function π . At each discrete time π‘:
β’ the agent observes state π π‘ β π;
β’ it chooses action π π‘ β π΄ (among the possible actions in state π π‘ ) and executes it; β’ it receives an immediate reward π π‘ = π (π π‘ , π π‘ ), that can be positive, negative or neutral; β’ the state changes to π π‘+1 = πΏ(π π‘ , π π‘ ).
Assuming that π π‘ and π π‘+1 only depend on current state and action, the agent's goal is to learn an action policy π : π β π΄ that maximizes the expected sum of (discounted) rewards obtained if policy π is followed. For each possible policy π the agent might adopt, we can define an evaluation function over states:
where π π‘ , π π‘+1 , ... are generated executing policy π starting at state π . Then, the choice of the best actions to play becomes an optimization problem. Indeed, it comes down to finding the optimal policy π * that maximizes (1) for all states π :
In the Q-learning framework, a numeric value π(π , π) β R, called Q-value, is associated to each state-action pair. The value of π is the reward received immediately upon executing action π from state π , plus the value (discounted by πΎ) of following the optimal policy thereafter:
where πΏ(π , π) denotes the state resulting from applying action π to state π . Then, we can reformulate (2) as:
The Q-values are estimated in the Q-learning algorithm by iterative Bellman updates:
In this way, if the agent learns the π function instead of π π * , it will be able to select optimal actions even if it has no knowledge of π and πΏ.
In this section, we define the operating environment in which the agent works.
Simulation environment: since it is difficult to retrieve real traffic data and perform real-world experimentation, we relied on SUMO [12], an open source traffic simulator that makes it possible to model real-world traffic behavior. This software, through an API called TraCI, provides complete control over the simulation environment elements, such as vehicles' speed and position, traffic flow's intensity on each lane, traffic light phases, the shape of the intersection, etc.
Performance measures: the performance of the agent is assessed with respect to two common traffic metrics: queue length and vehicles' waiting times. The goal is to find a model able to dynamically control the traffic lights of an intersection in order to minimize these two metrics.
Although dynamic traffic light control is an extremely complex task in the real world, SUMO allows you to operate in a more controlled environment. Specifically, the agent works in a fully-observable environment, since the software gives access to its complete state at each point in time. For this paper, we also defined a deterministic environment by setting a non-stochastic traffic flow generation. This makes the analysis simpler, but it is also one of the biggest limitations of this work. Clearly, the environment is also sequential and single agent.
In order to build a reinforcement learning model for traffic lights control, we need to define the traffic state representation, the action space and the reward function.
We propose a state representation that takes into consideration both vehicles' positions and waiting times. The idea is to map each lane approaching the intersection into a Boolean-valued vector, where each cell can contain a 1, indicating the presence of a vehicle at that position, or a 0, indicating its absence. Each cell of the vector corresponds to 1 meter of the lane. The matrix of vehicles' positions is then obtained by stacking all the lane vectors. Given an intersection with π lanes, where the longest lane is π meters, this intermediate state representation π β² consists of a (π Γ π) matrix. Note that a zero-padding is added to lane vectors with a length less than the maximum length lane in order to have all equally-sized vectors.
Then, the π β² representation is enriched by using a stack of consecutive simulation frames to make the model able to implicitly estimate vehicles' velocity and travel direction. In particular, π β² is computed for the last π (π = 2 in our setting) simulation steps, yielding a new (π Γ π Γ π) matrix, denoted as π β²β² .
The π β²β² representation built so far consists of a Boolean-valued matrix that contains the information about vehicles' positions of the last π simulation steps. However, it does not take into consideration the waiting times. This information is embodied in the representation by computing another state matrix, whose cells contain the normalized values of the vehicles' waiting times of the last simulation step. Then, the final state representation π is a (π + 1 Γ π Γ π) matrix.
To handle traffic at the intersection, the agent selects which lanes get a green light according to a set of three possible green/red light configurations. On each of the three incoming roads there is a traffic light that manages the traffic on the corresponding lanes. The combination of the individual phases of these traffic lights forms the set of the possible green/red light configurations. In Figure 1, all the three
The proposed definition of the reward function takes into account both the variation in queue length and waiting times. In particular, the reward π π‘ is given by the following formula:
where π½ π‘ represents the sum of the jam lengths (in meters) observed over the lanes at time π‘, and π π‘+1 represents the sum of the maximum waiting times (in seconds) observed over the lanes at time π‘ + 1. πΌ is a hyper-parameter that determines how much to penalize the agent for letting vehicles wait too much (in our setting πΌ = 0.4). The agent receives a positive reward if the last action performed, π π‘ , leads to a state π π‘+1 with shorter total queue length and/or low maximum waiting times.
The proposed architecture is a convolutional neural network that takes as input the state matrix mentioned in Section 5.1 and returns as output an approximation of the optimal Q-values. The model is composed of two convolutional layers and two fully connected layers at the end. In particular, the first convolutional layer consists of 16 (2 Γ 10)-filters with stride (2 Γ 1) followed by a LeakyReLU activation function. The second layer has 32 (1 Γ 4)-filters with stride (1 Γ 2) followed by a LeakyReLU activation function and a max pooling layer of size (1 Γ 2). The first fully-connected layer has 256 nodes followed by a LeakyReLU activation function, while the output layer has 3 linear output neurons (one for each possible green/red light configuration). In Figure 2, a summary of the CNN architecture is shown. We designed the convolutional kernels so that, ideally, they compute high-level representations of each road separately. Then, the joint information among the different roads is merged by the network in the last two fully connected layers. 1, which are typically found in literature.
The considered intersection (Figure 1) is composed of three incoming roads, each with two lanes. In order to simulate real-life scenarios, the intersection was designed similarly to a real one located in Como (IT) at the following coordinates: (45.802155, 9.084961). The two main roads' lengths are Select action π π‘ sampling from π ππ π‘πππ₯(π(π π‘ ; π))
Optimize π, using ADAM, according to πππ π 22:
Evaluate π 309π and 211π respectively, while the minor road's length is 103π. The max speed is 13.9π/π , which is equal to 50ππ/β, on each road. On each lane, vehicles can travel following different routes through the intersection. Due to the difficulty of finding a dataset of the traffic flows of Italian roads, we set the traffic flow rate to 450 vehicles per hour on each route. A scheme of the routes that vehicles can travel is shown in Figure 1. We can observe that the east incoming road has 4 different routes; therefore, the traffic on that road will be higher than on the others. The minimum green/red-light phase duration is fixed at 10 simulation steps (10 seconds in the simulation environment), while the yellow-light phase duration between two neighboring phases is fixed at 5 seconds. These two fixed lengths determine how many simulation steps SUMO can run before letting the model take a new action. With this configuration, the green-light phase is guaranteed to be of at least 10 seconds. For simplicity, we chose to generate only one vehicle's type. In particular, each vehicle's length is 5 meters. After 500 simulation steps, the system stops generating vehicles and the simulation ends. The proposed model was trained for 45 epochs, where each epoch is composed of 5 complete SUMO simulations.
As we said before, the proposed model was assessed with respect to two common traffic metrics: queue length and vehicles' waiting times. We compared the performance of the proposed model with that of the following baselines:
β’ A Multi-Layer Perceptron (MLP) network with one fully-connected hidden layer of 80 nodes, followed by a ReLU activation function, and 3 linear output neurons. The input of the MLP consists of a vector containing the information about the current phase, the queue length (in meters) on each lane, and the maximum time (in seconds) a vehicle has waited on each lane at the intersection, following the approach proposed in [6]. The MLP was trained with the same hyper-parameters and optimization method used for the CNN. β’ Two traffic control systems provided by default by SUMO: (1) the first one is a simple Static configuration of traffic light signals, in which all the phases cyclically change in a fixed sequence and each green/red-light phase has a fixed duration of 25 seconds, while the yellow-light duration is still 5 seconds. (2) The second system is the default implementation of the gap-based Actuated traffic control scheme, which dynamically adjusts traffic light phases' durations whenever a continuous stream of traffic is detected. β’ Two models that implement the most waiting first (MWF) heuristic and longest queue first (LQF)
heuristic. The first model sets a green light to lanes in which vehicles waited the most, up to the current simulation step. The second model, instead, sets a green light to lanes in which the longest queues were observed. For both models, the green/red-light duration and the yellow-light duration are the same as the CNN model. Table 2 shows the performance of the tested models. It is clear that the proposed agent performs better than every baseline, providing less average waiting time 1 and less average queue length. We can also see that the non-parametric methods such as MWF, LQF, Actuated and Static heuristics perform dramatically worse than the RL-based agents. Therefore, we continue the analysis by exploring the differences between the two neural network models.
In Figure 3, a comparison between the average rewards obtained by the CNN and the MLP on each epoch is shown (red line and blue line, respectively). The learning process seems to be more stable for the CNN-based agent, which performs better than the baseline. However, we can observe a rapid increase in the rewards obtained by the MLP agent at the end of the training. This suggests that, even if the CNN model provides better results in this experiment, the MLP does not perform dramatically worse. The same result can be deduced by looking at the average queue lengths and average waiting times obtained by the two architectures over the epochs, shown in Figure 4. For this reason, in order to 5 shows the box plots of the average rewards obtained by training both models in 4 different simulation setups, featuring increasing traffic intensities. Each setup is equivalent to the one presented in Section 6.1, with 350, 450, 550 and 700 vehicles per hour, respectively. The results show that, under low traffic conditions, the two models perform very similarly. However, the CNN-based agent scales better than the baseline with increasing traffic intensity, showing that the proposed model is more robust and can deal with more complex scenarios.
Smart cities and the planet urgently ask for environmental emissions to be reduced while improving the quality of life for citizens. To this end, Artificial Intelligence can provide researchers with instruments and tools to help this virtuous process. In this paper, a new CNN-based approach has been designed and tested to improve queue length and vehicle waiting times for traffic light control systems. The proposed approach has been extensively experimented against five baseline models. The results show that CNN models perform better than baselines. This opens the possibility of testing our approach in real-life conditions and in future Smart Cities that will exploit intelligent traffic light control systems.