-Traffic monitoring and control, as well as traffic simulation, are still significant and open challenges despite the significant researches that have been carried out, especially on artificial intelligence approaches to tackle these problems. This paper presents a Reinforcement Learning approach to traffic lights control, coupled with a microscopic agent-based simulator (Simulation of Urban MObility - SUMO) providing a synthetic but realistic environment in which the exploration of the outcome of potential regulation actions can be carried out. The paper presents the approach, within the current research landscape, then the specific experimental setting and achieved results are described.
Index Terms—reinforcement learning, traffic lights control, traffic management, agent-based simulation
Traffic monitoring and control and, in general, approaches supporting the reduction of congestion still represent hot topics for research of different disciplines, despite the substantial researches that have been devoted to these topics. The global phenomenon of urbanization (half of the world’s population was living in cities at the end of 2008 and it is predicted that by 2050 about 64% of the developing world and 86% of the developed world will be urbanized1) is in fact constantly changing the situation and making it actually harder to manage such a concentration of population and transportation demand. Technological developments among which autonomous driving represents just the most futuristic one (at least from a popular culture perspective), represent at the same time attempts to tackle these issues and further challenges, in terms of potential developments whose introduction requires further study and analysis of the potential impact and implications.
Artificial Intelligence plays an important role within this
framework; even not considering the obvious relevance to the
autonomous driving initiative, we focus here on two aspects:
(i) the regulation of traffic patterns, especially based on (ii) the
analysis of situations by means of agent-based simulations, in
which the behaviour of drivers and other relevant entities is
modeled and computer within a synthetic environment. The
latter, in particular, have reached a level of sufficient
complexity, flexibility, and they have proven their capability to support
decision makers in the exploration of alternative ways to
manage traffic within urban settings. On the side of regulation
of traffic patterns, the availability of these simulators, coupled
with advances in machine learning, represents an opportunity
for a scientific investigation of the possibility to employ
these virtual environments as tools to explore the outcome of
potential regulation actions within specific situations, within a
Reinforcement Learning [
This paper represents a contribution within this line of
research and, in particular, we focus on a simple yet still
studied situation: a single four way intersection regulated by
traffic lights, that we want to manage through an autonomous
agent perceiving the current traffic conditions, and exploiting
the experience carried out in simulated situations, possibly
representing plausibile traffic conditions. The simulations are
actually also agent-based, and in particular, for this study,
they have been carried out in a tool for Simulation of Urban
MObility (SUMO) [
The paper breaks down as follows: we first provide a compact description of the relevant portion of the state of the art in traffic lights management with RL approaches, then we introduce the experimental setting we adopted for this study. The RL approach we defined and adopted will be given in Section IV, then the achieved results will be described. Conclusions and future developments will end the paper.
One of the acceptations of the goals of AI is to develop
machines that resemble the intelligent behavior of a human
being. In order to achieve this goal, an AI system should
be able to interact with the environment and learn how
to correctly act inside it. An established area of AI that
has been proved capable of experience-driven autonomous
learning is reinforcement learning [
Traffic signal control is a well suited application context for
RL techniques: in this framework, one or more autonomous
agents have the goal of maximizing the efficiency of traffic
flow that drives through one or more intersection controlled
by traffic lights. The use of RL for traffic signal control is
motivated by several reasons [
RL techniques applied to traffic signal control address the
following challenges: [
Although the above are potential advantages of the RL approach to traffic signal control, not all of them have already been achieved, and (as we will show in the remander of the paper) the present approach only represents an initial step in this overall line of work.
In order to apply a RL algorithm, it is necessary to define the state representation, the available actions and the reward functions; in the following, we will describe the most widely adopted approaches for the design of these elements within the context of Traffic Signal Control.
1) State representation: The state is the agent’s perception of the environment in an arbitrary step. In literature, state space representations particularly differ in information density.
In low information density representations, usually the
intersection’s lanes are discretized in cells along the length of the
lane. Lane cells are then mapped to cells of a vector, which
marks 1 if a vehicle is inside the lane cell, 0 otherwise [
Regarding state representations with high information
density, usually the agent receives an image of the current situation
of the whole intersection, i.e. a snapshot of the simulator being
used; multiple successive snapshots will be stacked together
to give the agent a sense of the vehicle motion [
2) Actions representation: In the context of traffic signal control, the agent’s actions are implemented with different degrees of flexibility and they are described below.
Among the category of action set with low flexibility, the
agent can choose among a defined set of light combinations.
When an action is selected, a fixed amount of time will
lasts before the agent can select a new configuration [
3) Reward representation: The reward is used by the agent to understand the effects of the latest action taken in the latest state; it is usually defined as a function of some performance indicator of the intersection efficiently, such as vehicles’ delays, queue lengths, waiting times or overall throughput.
Most of the works include the calculation of the change
between cumulative vehicle delay between actions, where the
vehicle delay is defined as the number of seconds the vehicles
is steady [
The most recent reinforcement learning research has proposed multiple possible solutions to address the traffic signal control problem, in which it emerges that different algorithms and neural networks structure can be used, although some common techniques are necessary but not sufficient in order to ensure a good performance.
The most widely used algorithm to address the problem is
Q-learning. The optimal behavior of the agent is achieved with
the passage of time is represented in simulation steps. But the
agent only operates at certain steps, after the environment has
evolved enough. Therefore, in this paper every step dedicated
to the agent’s workflow is called agentstep, while the steps
dedicated to the simulation are simply called ”steps”. Hence,
after a certain amount of simulation steps, the agent starts
its sequence of operations by gathering the current state
of the environment. Also, the agent calculates the reward
of the previous selected action, using some measure of the
current traffic situation. The sample of data containing every
information about the latest simulation steps is saved to a
memory and later extracted for a training session. Now the
agent is ready to select a new action based on the current
state of the environment, which will resume the simulation
until the next agent interaction.
the use of neural networks to approximate Q-values given a
state. Often, this approach includes a Convolutional Neural
Network (CNN) to compute the environment state and learn
features from an image [
Genders and Ravi [
Mousavi et al. [
In [
The traffic microsimulator used for this research is
Simulation of Urban MObility (SUMO) [
In this research, the chance of improvement in traffic flow that drives through an intersection controlled by traffic lights will be investigated using artificial intelligence techniques. The agent is represented by the traffic light system that interacts with the environment in order to maximize a certain measure of traffic efficiency. Given this general premise, the problem A. Training setup and traffic generation tackled in this paper is defined as follows: given the state of The entire training is divided in multiple episodes .The total the intersection, what is the traffic light phase that the agent number of episodes is 300. By default, SUMO provides a time should choose, selected from a fixed set of predefined actions, frequency of 1 second per step, and the period of each episode in order to maximize the reward and consequently optimize is set at 1 hour and 30 minutes, therefore the total number of the traffic efficiency of the intersection. steps per episode is equal to 5400. 300 episodes of 1.30 hours
The typical workflow of the agent is shown in Figure 2. each are equivalent to almost 19 days of continuous traffic, and It should be underlined that in this application with SUMO, the entire training takes about 6 hours on a high-end laptop.
The environment where the agent acts is represented in Figure 3. It is a 4-way intersection where 4 lanes per arm approach the intersection from the compass directions, leading to 4 lanes per arm leaving the intersection. Each arm is 750 meters long. On every arm, each lane defines the possible directions that a vehicle can follow: the right-most lane enable vehicles to turn right or going straight, the two central lanes bound the driver to go straight while on the left-most lane the left turn is the only direction allowed. In the center of the intersection, a traffic light system, controlled by the agent, manages the approaching traffic. In particular, on every arm the left-most lane has a dedicated traffic light, while the other three lanes share a traffic light. Every traffic light in the environment operates according to the common european regulations, with the only exception being the absence of time between the end of a yellow phase and the start of the next green phase. In this environment pedestrians, sidewalks and pedestrian crossings are not included.
In a simulated intersection, the traffic generation is a crucial part that can have a big impact on the agents performance. In order to maintain a high degree of reality, in each episode the traffic will be generated according to a Weibull distribution with a shape equal to 2. An example is shown in Figure 4.
The distribution is presented in the form of a histogram, where the steps of one simulation episode are defined on the x-axis and the number of vehicles generated in that step window is defined on the y-axis. The Weibull distribution approximates specific traffic situations, where during the early stage the number of cars is rising, representing a peak hour. Then, the number of incoming cars slowly decreases describing the gradual mitigation of traffic congestion. Also, every vehicles generated has the same physical dimensions and performance.
The traffic distribution described provides the exact step
of the episode when a vehicle will be generated. For every
vehicle scheduled, its source arm and destination arm are
determined using a random number generator which have a
different seed in every episode, so it is not possible to have two
equivalent episodes. In order to obtain a true adaptive agent,
the simulation should include a significant variety of traffic
flows and patterns [
• High-traffic scenario. 4000 cars approach the intersection from every arm evenly distributed. Then, 75 % of generated cars will go straight and 25 % of cars will turn left or right at the intersection. • Low-traffic scenario. 600 cars approach the intersection from every arm evenly distributed. Then, 75 % of generated cars will go straight and 25 % of cars will turn left or right at the intersection. • NS-traffic scenario. 2000 cars approach the intersection, with 90 % of them coming from the North or South arm.
Then, 75 % of generated cars will go straight and 25 % of cars will turn left or right at the intersection. • EW-traffic scenario. 2000 cars approach the intersection, with 90 % of them coming from the East or West arm.
Then, 75 % of generated cars will go straight and 25 % of cars will turn left or right at the intersection.
Each scenario corresponds to one single episode and they cycle during the training always in the same order.
IV. DESCRIPTION OF THE REINFORCEMENT LEARNING
APPROACH
In order to design a system based on the reinforcement learning framework, it is necessary to define the state representation, the action set, the reward function and the agent learning techniques involved. It should be noted that the such agent’s elements in this paper are easily replaceable with a traffic monitoring system in a real world appliance, compared to others relevant studies in this topic which have higher requirements in terms of technical feasibility.
The state of the agent describes a representation of the situation of the environment in a given agentstep t and it is usually denoted with st. To allow the agent to effectively learn to optimize the traffic, the state should provide sufficient information about the distribution of cars on each road.
The objective of the chosen representation is to let the
agent knows the position of vehicles inside the environment
at agentstep t. For this purpose the approach proposed in this
paper is inspired to the DTSE [
Technically, in each arm of the intersection incoming lanes are discretized in cells that can identify the presence or absence of a vehicle inside them. In Figure 5 is showed the state representation for the west arm of the intersection. Between the beginning of the road and the intersection’s stop line, there are 20 cells. 10 of them are located along the left-only lane while the others 10 cover the others three lanes. Therefore, in the whole intersection there are 80 cells. Not every cell has the same size: the further the cell is from the stop line, the longer it is, so more lane length is covered. The choice of the length of every cell is not trivial: if cells were too long, some cars approaching the crossing line may not be detected; if cells were too short, the number of states required to cover the length of the lane increases, bringing to higher computational complexity. In this paper, the length of the shortest cells, which are also the closest to the stop line, is exactly 2 meters longer than the length of a car.
In summary, whenever the agent observe the state of the environment, he will obtain the set of cells that describe the presence or absence of vehicles in every area of the incoming roads.
The action set identifies the possible actions that the agent can take. The agent is the traffic light system, so doing an action translates to activate a green phase for a set of lanes for a fixed amount of time, choosing from a predefined set of green phases. In this paper, the green time is set at 10 seconds and the yellow time is set at 4 seconds. Formally, the action space is defined in the set (1). The set includes every possible action that the agent can take.
A = {NSA, NSLA, EWA, EWLA} (1) Every action of set (1) is described below. • North-South Advance (NSA): the green phase is active for vehicles that are in the north and south arm and wants to proceed straight or turn right. • North-South Left Advance (NSLA): the green phase is active for vehicles that are in the north and south arm and wants to turn left. • East-West Advance (EWA): the green phase is active for vehicles that are in the east and west arm and wants to proceed straight or turn right. • East-West Left Advance (EWLA): the green phase is active for vehicles that are in the east and west arm and wants to turn left.
Figure 6 shows a graphical representation of the four possible actions.
If the action chosen in agentstep t is the same as the action taken in the last agentstep t − 1 (i.e. the traffic light combination is the same), there is no yellow phase and therefore the current green phase persists. On the contrary, if the action chosen in agentstep t is not equal to the previous action, a 4 seconds yellow phase is initiated between the two actions. This means that the number of simulation steps between two same actions is 10, since 1 simulation step is equal to 1 second in SUMO. When the two consecutive actions are different, the yellow phase counts as 4 extra simulation steps and therefore the total number of simulation steps in between actions is 14. Figure 7 shows a brief scheme of this process.
In reinforcement learning, the reward represents the
feedback from the environment after the agent has chosen an
action. The agent uses the reward to understand the result
of the taken action and improve the model for future action
choices. Therefore, the reward is a crucial aspect of the
learning process. The reward usually has two possible values:
positive or negative. A positive reward is generated as a
consequence of good actions, a negative reward is generated
from bad actions. In this application, the objective is to
maximize the traffic flow through the intersection over time. In
order to achieve this goal, the reward should be derived from
some performance measure of traffic efficiency, so the agent
is able to understand if the taken action reduce or increase the
intersection efficiency. In traffic analysis, several measures are
used [
called literature because it is inspired to similar studies in this topic. The literature reward function uses as a metric the total waiting time, defined as in equation (2).
n X veh=1 twtt = wt(veh,t) (2) Where wt(veh,t) is the amount of time in seconds a vehicle veh has a speed of less than 0.1 m/s at agentstep t. n represents the total number of vehicles in the environment in agentstep t. Therefore, twtt is the total waiting time at agentstep t. From this metric, the literature reward function can be defined as a function of twtt and is shown in (3) rt = 0.9 · twtt−1 − twtt (3) Where rt represents the reward at agentstep t. twtt and twtt−1 represent the total waiting time of all the cars in the intersection captured respectively at agentstep t and t − 1. The parameter 0.9 helps with the stability of the training process.
In a reinforcement learning application, the reward usually can be positive or negative, and this implementation is no exception. The equation 3 is designed in such a way that when the agent chooses a bad action it returns a negative value and when it chooses a good action it returns a positive value. A bad action can be represented as an action that, in the current agentstep t, adds more vehicles in queues compared to the situation in the previous agentstep t − 1, resulting in higher waiting times compared to the previous agentstep. This behavior increases the twt for the current agentstep t and consequently the equation 3 assumes a negative value. The more vehicles were added in queues for the agentstep t, the more negative rt will be and therefore the worst the action will be evaluated by the agent. The same concept is applied for good actions.
The problem with this reward function lays inside the choiche of the metric, and happens when the following situation arise. During the High-traffic scenario, very long queues appears. When the agent activate the green phase for a long queue, the departure of cars creates a wave of movement that traverse the entire queue. The reward associated to this phase activation is received not only in the next agentstep, as it should, but also in very next ones. That is because the movement wave persists longer compared to the delta step between actionstep, and the wave triggers the waiting times of cars in the queue, misleading the agent about the reward received.
function uses a metric that is slightly different from the former metric, which is the accumulated total waiting time, defined in equation (4).
atwtt = n X awt(veh,t) veh=1 (4) Where awt(veh,t) is the amount of time in seconds a vehicle veh has a speed of less than 0.1 m/s at agentstep t, since the spawn into the environment. n represents the total number of vehicles in the environment in agentstep t. Therefore, atwtt is the accumulated total waiting time at agentstep t. With this metric, when the vehicle departs but it does not manage to cross the intersection, the value of atwtt does not resets (unlike the value of twtt), avoiding the misleading reward associated with the literature reward function, when a long queue build up at the intersection. Once the metric is set, the alternative reward function is defined such as in equation (5) the cars in the intersection captured respectively at agentstep t and t − 1.
The learning mechanism involved in this paper is called
Deep Q-Learning, which is a combination of two aspects
widely adopted in the field of reinforcement learning: deep
neural networks and Q-Learning. Q-Learning [
Q(st, at) = Q(st, at)+α(rt+1+γ·maxAQ(st+1, at)−Q(st, at)) (6) where Q(st, at) is the value of the action at taken from state st. The equation consists on updating the current Q-value with a quantity discounted by the learning rate α. Inside the parenthesis, the term rt+1 represents the reward associated to taking action at from state st. The subscript t + 1 is used to emphasize the temporal relationship between taking the action at and receiving the consequent reward. The term Q(st+1, at) represents the immediate future’s Q-value, where st+1 is next state in which the environment has evolved after taking action at in state st. The expression maxA means that, among the possible actions at in state st+1, the most valuable is selected. The term γ is the discount factor that assumes a value between 0 and 1, lowering the importance of future reward compared to the immediate reward.
In this paper, a slightly different version of the equation (6) is used and it is presented in equation (7). This will be called the Q-learning function from this point.
′ Q(st, at) = rt+1 + γ · maxAQ (st+1, at+1) (7) Where the reward rt+1 is the reward received after taking action at in state st. The term Q′(st+1, at+1) is the Q-value associated with taking action at+1 in state st+1, i.e. the next state after taking action at in state st. As seen in equation (6), the discount factor γ denote a small penalization of the future reward compared to the immediate reward. Once the agent is trained, the best action at taken from state st will be the one that maximize the function Q(st, at). In other words, maximizing the Q-learning function means following the best strategy that the agent have learned.
In a reinforcement learning application, often the state space
is so large that is impractical to discover and save every
stateaction pair. Therefore, the Q-learning function is approximated
using a neural network. In this paper, a fully connected deep
neural network is used, which is composed of an input layer of
80 neurons, 5 hidden layers of 400 neurons each with rectified
linear unit (ReLU) [
Where rt represents the reward at agentstep t. atwtt and Experience replay [
A sample m is formally defined as the quadruple (8).
m = {st, at, rt+1, st+1}
(8) Where rt+1 is the reward received after taking the action at from state st, which evolves the environment into the next state st+1. This technique is implemented to remove correlations in the observation sequence, since the state of the environment st+1 is a direct evolution of the state st and the correlation can decrease the training capability of the agent. In Figure 9 is shown a representation of the data collection task.
Fig. 9. Scheme of the data collection.
As stated earlier, the experience replay technique needs a memory, which is characterized by a memory size and a batch size. The memory size represents how many samples the memory can store and is set at 50000 samples. The batch size is defined as the number of samples that are retrieved from the memory in one training instance and is set at 100. If at a certain agentstep the memory is filled, the oldest sample is removed to make space for the new sample.
A training istance consists of learning the Q-value function iteratively using the information contained in the batch of samples extracted. Every sample in the batch is used for training. From the standpoint of a single sample, which contains the elements {st, at, rt+1, st+1}, the following operations are executed: 1) Prediction of the Q-values Q(st), which is the current knowledge that the agent has about the action values from st. 2) Prediction of the Q-values Q′(st+1). These represents the knowledge of the agent about the action values starting from the state st+1. 3) Update of Q(st, at) which represents the value of the particular action at selected by the agent during the simulation. This value is overwritten using the Q-learning function described in equation (7). The element rt+1 is the irsewoabrtdainasesdocuisaitnegd tthoethpereadcitcitoionnato,f mQa′(xsAt+Q1′)(satn+d1,raetp+r1e)sents the maximum expected future reward i.e. the higher action value expected by the agent, starting from state st+1. It will be discounted by a factor γ that gives more importance to the immediate reward. 4) Training of the neural network. The input is the state st, while the desired output is the updated Q-values Q(st, at) that now includes the maximum expected future reward thanks to the Q-value update.
Once the deep neural network has sufficiently approximated the Q-learning function, the best traffic efficiency is achieved by selecting the action with the highest value given the current state. A major problem in any reinforcement learning task is the action-selection policy while learning; whether to take exploratory action and potentially learn more, or to take exploitative action and attempt to optimize the current knowledge about the environment evolution. In this paper the ǫ-greedy exploration policy is chosen, and it is represented by the equation (9). It defines a probability ǫ for the current episode h to choose an explorative action, and consequently a probability 1 − ǫ to choose an exploitative action.
(9) ǫh = 1 − h
H where h is the current episode of training and E is the total number of episodes. Initially, ǫ = 1, meaning that the agent exclusively explores. However, as training progresses, the agent increasingly exploits what it has learned, until it exclusively exploits.
The performance of the agents is assessed in two parts: initially, the reward trend during the training is analyzed. Then, a comparison between the agents and a static traffic light is discussed, with respect to common traffic metrics, such as cumulative wait time and average wait time per vehicle.
One agent is trained using the literature reward function, while the other one adopts the alternative reward function. Figure 10 shows the learning improvement during the training in the Low-traffic scenario of both agents, in term of cumulative negative reward i.e the magnitude of actions’ negative outcomes during each episode. As it can be seen, each agent has learned a sufficiently correct policy in the Low-traffic scenario. As the training proceeded, both agents efficiently explore the environment and learn an adequate approximation of the Q-values; then, towards the end of the training, they try to optimize the Q-values by exploiting the knowledge learnined so far. The fact that the agent with the alternative reward function has a better reward curve overall is not a strong evidence of a better performance, since haveing two different reward functions means that different reward values are produced. The performance difference will be discussed later during the static traffic light benchmark.
Figure 11 shows the same training data as Figure 10, but referred to the High-traffic scenario. In this scenario, the agent with the literature reward shows a significantly unstable reward curve, while the other agent’s trend is stable. This behavior is caused by the choiche of using the waiting time of vehicles as a metric for the reward function, which in situations with long queues causes the aquisition of misleading rewards. In fact, by using the accumulated waiting time like in the alternative reward function, vehicles does not resets their waiting times by simply advancing through the queue. As Figure 11 shows, the alternative reward function produces a more stable policy. In the NS-traffic and EW-traffic scenarios, both agents perform well since it is a simpler task to exploit.
In order to truly analyze which agent achieve better
performance, a comparison between the agents and a Static Traffic
Light (STL) is presented. The STL has the same layout of
the agents and it cycle through the 4 phases always in the
following order: [NSA − NSLA − EWA − EWLA]. Moreover,
every phase has a fixed duration and they are inspired by those
on real-world static traffic lights [
In Table I are shown the performance of the two agents, compared to the STL. The metric used to measure the performance difference are the cumulative wait time and the average wait time per vehicle. The cumulative wait time is defined as the sum of all waiting times of every car during the episode, while the average waiting time per vehicle is defined as the average amount of seconds spent by a vehicle in a steady position during the episode. These measures are gathered across 5 episodes and then averaged.
cwt awt/v cwt awt/v cwt awt/v cwt
Literature reward Alternative reward agent agent
Low-traffic scenario -30 -29 +145 +136 -50 -47 -65
High-traffic scenario NS-traffic scenario EW-traffic scenario -47 -45 +26 +25 -62 -56 -65
In general, the alternative reward agent achieves a better traffic efficiency compared to the literature agent: this is a consequence of the adoption of a reward function (accumulated waiting time) that more properly discounts waiting times exceeding a single traffic light cycle. Considering just the waiting time starting from the last stop of the vehicles, leads to not sufficiently emphasize the usefulness of keeping longer light cycles, introducing too many yellow lights situations and changes, that are effective in low or medium traffic situations. The fact that the agent is more effectine in low to medium traffic situations, leads to think that an easy and almost immediate opportunity would be to separately develop agents devoted to different traffic situations, having a sort of controller that monitors the traffic flow and that selects the most appropriate agent configuration. This experimentation also leads to consider that, however, additional improvements would be possible by (i) improving the learning approach to achieve a more stable and faster convergence, (ii) further improving the reward fuction to better describe the desired behaviour and to influence the average cycle lengths, that is more fruitfully short in low traffic situation and long whenever the traffic condition worsens.
This work has presented a believable exploration of the plausibility of a RL approach to the problem of traffic lights adaptation and management. The work has employed a realistic and validated traffic simulator to provide an environment in which training and evaluating a RL agent. Two metrics for the reward of agent’ actions have been investigated, clarifying that a proper decription of the application context is just as important as the competence in the proper application of machine learning approaches for achieving proper results.
Future works are aimed at further improving achieved results, but also, within a longer term, at investigating what would be the implications of introducing mutiple RL agents within a road network and what would be the possiblity to coordinate their efforts for achieving global improvements over local ones, and also the implications on the vehicle population, that could perceive the change in the infrastructure and adapt in turn to exploit additional opportunities and potentially negating the achieved improvements due to an additional traffic demand on the improved intersections. It is important to perform analyses along this line of work to understand the plausibility, potential advantages or even unintended negative implications of the introduction in the real world of this form ofself-adaptive system.