Introduction

Reinforcement Learning (RL) for Traffic Signal Control (TSC) is an area which has been investigated in detail as a potential improvement on the current adaptive systems in use. Current commercially available systems do not use RL, and require manual setup of signal timings for each intersection, something which can be time-consuming to do and can have a negative impact on traffic flow if not configured correctly. In the UK, MOVA [1] (Microprocessor Optimised Vehicle Actuation) and SCOOT [2] (Split, Cycle, and Offset Optimisation Technique) are the most widely implemented commercial systems, with the latter being used mainly for regions of up to 30 traffic signal junctions. While adaptive (extending green signals when traffic demand is high in a given direction), these algorithms do not use RL to learn the specific characteristics of a traffic signal. The design of these algorithms was completed in the 1980s, and the iterative improvements made since then have not taken advantage of the vast amount of information available now from roadside sensors. In addition to this, modern approaches to the TSC problem can employ more advanced data sources such as traffic cameras, and information from connected and autonomous vehicles, allowing for a more accurate picture of the traffic flow through a road network. Furthermore, this allows for prioritisation of certain types of traffic, where appropriate, such as allowing heavy goods vehicles (HGVs) to pass through lights and avoid deceleration (followed by acceleration), or clearing the road network in a certain direction to allow for easier passage of emergency vehicles attending to an emergency.

RL based approaches for TSC, whilst not exposed to the decades of development which current approaches in use have had, have still been shown to outperform well-calibrated systems in simulations [3]. Current state-of-the-art approaches make use of some innovative methods such as junction pressure [4,5], convolutional neural networks [6] and graph attention networks [7].

Introducing independent RL agents at each intersection within a road network has a number of benefits. Firstly, it allows for easier scalability when compared to a centralised system as changes to the road network such as the addition of new roads or traffic signals can be tolerated by introducing new agents, rather than re-training or modifying a central system. Secondly, the state and action space of a centralised agent increases exponentially when more traffic signals are introduced, leading to the curse of dimensionality [8]. Independent RL agents deployed to each intersection suffer from neither of these problems, and each agent can learn the specific characteristics of the intersection under their control. However, a problem emerges when we consider the simultaneous learning process which is used to train the independent agents. As an agent updates their policy to be optimal from their observations, the optimal policy for the agents at connected intersections from this agent may change based on the impact to the demand profile of their intersection. We refer to this as the problem of non-stationarity.

In this paper, we evaluate reward functions from the literature and review them in the context of a real-world multi-agent scenario, using calibrated data from Ingolstadt, Germany, to test them, including an implementation of a commonly used commercial solution, MOVA, as the baseline. We highlight the impact of reward functions on the ability of the agent to learn, and how solutions to the problem of non-stationarity may not be feasible in the real world when used in the TSC context. To evaluate the performance of different reward functions, we compare the waiting time and trip duration of vehicles.

Background and Related Work

The non-stationary problem is one which has been observed in many multi-agent RL contexts [9,10,11]. We define the non-stationary problem as when independent agents are in an environment where they take actions to optimise their policy, aided by a reward function, but the actions of these agents impact the surrounding agents. The changing environment can be referred to as non-stationary. When thought of in the context of the TSC problem, we encounter a changing environment when agents change their policies to one which they believe is more optimal. This change can impact the demand characteristics which other agents see, and may result in their own policies no longer being optimal. Furthermore, in addition to the reason of changing road networks, as well as the curse of dimensionality, it is also not feasible to have a centralised system to learn and control traffic timings as computational complexity exponentially grows in the numbers of lanes and junctions [12].

A potential solution to this problem is to employ an actor-critic (AC) algorithm [13] for each agent, with a common critic. In the context of TSC, multi-agent AC and the derivatives have been implemented and tested, with Feudal AC [14] evaluated by Ault et al. [6], and their investigation found that they perform similarly to Deep Q-Learning algorithms but take significantly longer to converge on the solution. An alternative approach to the problem of non-stationarity is to introduce a form of communication between agents. Foerster et al. [15] introduce a Deep Distributed Recurrent Q-Network, where agents share hidden layers and are tasked with developing a communication protocol to expedite the solving of communicationbased coordination tasks. Sukhbaatar et al. [16] introduced the architecture of CommNet, which incorporates a communication message, the average of the previous hidden layers from all other agents into the input of each layer of the agent. However, for both AC approaches and communication between agents, the issues around scalability remain, and may require the critic or communicative agent to be retrained when changes are made to the road network.

In work by Cabrejas-Egea et al. [17], an assessment of 15 common reward functions, aggregated into 5 groups (queue-length based rewards, waiting time based rewards, delay based rewards, average speed based rewards and throughput based rewards), is performed and it is found that average speed maximisation reduces the average vehicle waiting time. However, this was performed in a single agent scenario, with one junction. Whilst maximising speed may perform best in isolated junctions, it is unknown how nearby junctions will be affected. Wei et al. [18] provides more details on alternative approaches in RL for TSC, including the state and reward functions employed and the dataset used to verify results.

Moreover, it is suggested that there is a significant gap between the performance of agents in synthetic benchmarks and calibrated data from the real world. Ault et al. [6] compared implementations of MPLight [19], FMA2C [14] and DQN based approaches [20] (among others) and concluded that whilst synthetic benchmarks can prove challenging for RL agents, there is a difference in performance between them and calibrated data. There is a gap in the literature to explore whether this continues into reward functions as well. Specifically, we are interested in the performance of reward functions in realistic traffic scenarios.

Problem Formulation

The TSC problem can be formulated as a Partially Observable Markov Decision Process (POMDP) [21] < 𝑆, 𝐴, ℙ, 𝑅, Ω, 𝑂, 𝛾 >, defined as 𝑆, the set of states, 𝐴, the set of possible actions, ℙ(𝑠 𝑡 , 𝑎, 𝑠 𝑡+1 ) ∶ 𝑆 × 𝐴 × 𝑆 → [0, 1], the state transition function, 𝑅(𝑠, 𝑎, 𝑠 ′ ) ∶ 𝑆 × 𝐴 → ℝ which describes the likelihood of transitioning from 𝑠 to 𝑠 ′ when action 𝑎 is taken, Ω, the set of observations, 𝑂, the set of conditional observation probabilities 𝑂 ∶ 𝑆 × 𝐴 × Ω → [0, 1], and 𝛾, the discount factor. We define this problem as a POMDP rather than a standard Markov Decision Process due to the limitations in sensor capability and knowledge of the global state. Therefore, 𝑆 can be defined as the state of the system, contrasted to Ω, the observations of the system state from the sensors at an intersection.

The choice of reward function 𝑅 is important to the performance of our agents. In the TSC problem, the high-level aim is to maximise throughput of vehicular traffic across all traffic signals. Part of increasing throughput is to reduce vehicle waiting time and increase average speed as these two factors directly contribute to how quickly vehicles reach their destination. However, for the same reasons that it is not feasible to use a centralised single agent to control all the intersections, it is not feasible to incorporate the total throughput of all agents as a reward function. Furthermore, the problem of non-stationarity is still prevalent as the reward that agents see will now be explicitly and directly impacted by the policies of other agents.

In the context of TSC, we define a phase 𝜑 as a group of non-conflicting green lights at a signalised intersection, and a signalised intersection as having a finite set of phases Φ such that 𝜑 ∈ Φ. In each intersection, we construct the state space (𝑆) as a combination of the current phase the intersection has selected and the observation of the current traffic state. In addition to this, we can define the action space (𝐴) for an agent as Φ. If the selected phase is a change to the current phase, there must be a mandatory yellow phase interjected, and the selected phase must also be chosen for longer than the minimum limit [22]. Each intersection includes an emulated traffic signal controller, and if an agent selects a different phase or an action which does not fulfill the mandatory requirements, the traffic controller enforces the legal safety requirements. The reward function 𝑅 differs between implementations, and how to choose this is the focus of our paper.

It should also be mentioned that by describing the TSC problem as an POMDP, we are assuming that the TSC problem fulfils the Markov property, that is, that the process of TSC is memory-less (the result of the next state only depends on the action taken from the current state). Formally, given a state history 𝑆 𝐻 :

𝑆 𝐻 = 𝑆 𝑡 , 𝑆 𝑡+1 , ..., 𝑆 ∞(1)

then, if following the Markov Property

ℙ(𝑆 𝑡+1 |𝑆 𝐻 ) = ℙ(𝑆 𝑡+1 |𝑆 𝑡 , 𝑆 𝑡+1 , ..., 𝑆 ∞ ) = ℙ(𝑆 𝑡+1 |𝑆 𝑡 )(2)

When applied to the context of TSC, it may seem like this assumption does not hold true, as traffic has known periodic cycles of greater and lesser demand. Son et al. [23] showed that fluctuations in traffic flow (and seasonality) can be modelled using a Fourier Transform, and used to make predictions about future traffic predictions. However, this is only possible when the states of traffic signals are viewed over a period of days to weeks, and in the TSC problem, this temporal horizon is very small (seconds to minutes) and in a resolution below the required amount to make assumptions regarding traffic seasonality. With the assumption that TSC does fulfil the Markov property, and taking into consideration the computational complexity of solving POMDP, we model the problem as a regular MDP.

When RL is applied to this MDP, the aim of the agents is to learn a policy 𝜋 to maximise the future discounted reward defined by:

∞ ∑ 𝑡=0 𝛾 𝑡 𝑅(𝑠 𝑡 , 𝑎 𝑡 )(3)

Where 𝛾 ∈ [0, 1]. Q-learning, an off-policy model-free value-based RL algorithm is an effective and powerful tool in solving MDPs and has been shown to find an optimal policy (one which maximises expected total discounted reward) in any finite MDP [24]. This approach aims to learn an optimal action-value (Q) function 𝑄 * (𝑠, 𝑎) given a state 𝑠 and action 𝑎: when optimal policy 𝜋 * is followed.

𝑄 * (𝑠, 𝑎) = 𝔼[𝑟|𝑠, 𝑎] + 𝛾 ∑ 𝑠 ′ ℙ(𝑠 ′ |𝑠, 𝑎) max 𝑎 ′ 𝑄 * (𝑠 ′ , 𝑎 ′ )(4)

Q-learning, in this format, takes the form of a table-based algorithm which recursively approximates 𝑄 * (𝑠, 𝑎) through iterative Bellman updates with a learning rate of 𝛼 and temporal difference target of 𝑦 𝑡 for the Q-function:

𝑄 * (𝑠 𝑡+1 , 𝑎 𝑡+1 ) ← 𝑄 𝜋 (𝑠 𝑡 , 𝑎 𝑡 ) + 𝛼(𝑦 𝑡 − 𝑄(𝑠 𝑡 , 𝑎 𝑡 )) 𝑦 𝑡 = 𝑅 𝑡 + 𝛾 max 𝛼 𝑡+1 𝑄 𝜋 (𝑠 𝑡+1 , 𝑎 𝑡+1 )(5)

A major improvement to Q-learning performance came from using a convolutional neural network for the Q-value estimator combined with a novel experience replay mechanism and an iterative periodic update process which allowed the Deep Q-Network (DQN) agent to converge on an optimal policy when tested on the Atari 2600 dataset [25].

Reward Functions

The following functions are experimentally reviewed. We review reward functions from the literature (1,3,4,6,8,11) and propose some functions here (2, 3, 5, 7, 9, 12), inspired by the previously proposed algorithms. We define 𝑉 𝑡 as the set of vehicles in incoming lanes and 𝑚 𝑣 as as the speed of vehicle 𝑣. Furthermore, we define 𝜏 𝑣 as the waiting time of vehicles. Similar to the definition of upstream traffic 𝑉 𝑡 , we define pressure as 𝑃 𝑥 where 𝑥 ∈ {𝑢𝑝, 𝑑𝑜𝑤𝑛}, and {𝑢𝑝, 𝑑𝑜𝑤𝑛} representing the upstream and downstream traffic flows respectively.

1. Average Speed: Used in [26], we aim for the agent to maximise the flow of vehicles by reducing the amount of time stopped or at low speeds. This is the optimal solution proposed by [17].

𝑟 𝑡 = 1 |𝑉 𝑡 | ∑ 𝑣∈𝑉 𝑡 𝑚 𝑣(6)

2. Average Speed Normalised: By normalising the average speed with the maximum observed speed in a lane 𝑚 𝑚𝑎𝑥 (defined as max 𝑉 𝑡 (𝑚 𝑣 )), we aim to reduce any problems caused by different speed limits in the approaches to the junction.

𝑟 𝑡 = 1 |𝑉 𝑡 | ∑ 𝑣∈𝑉 𝑡 𝑚 𝑣 𝑚 𝑚𝑎𝑥(7)

3. Maximum Wait Time: This approach prioritises the vehicles which have been waiting the longest.

𝑟 𝑡 = − max {𝑣∈𝑉 𝑡 } 𝜏 𝑡(8)

4. Aggregate Wait Time: As suggested by [27] ,the reward is the negative sum of the wait time of all the queuing cars.

𝑟 𝑡 = − ∑ 𝑣∈𝑉 𝑡 𝜏 𝑡(9)

5. Aggregate Wait Time Normalised: Similar to Aggregate Wait Time, but we use the maximum waiting time to normalise the value. This is so the agent is not forced into acting in a first in, first out manner which may happen with just using Aggregate Wait Time.

𝑟 𝑡 = − ∑ 𝑣∈𝑉 𝑡 𝜏 𝑡 𝜏 𝑚𝑎𝑥(10)

6. Pressure: Used in [4,5,28,29,19], pressure is a very common reward function, and is defined as the difference of vehicle density in the upstream lanes 𝑃 𝑢𝑝 and downstream lanes 𝑃 𝑑𝑜𝑤𝑛 . This approach has been promising simulations which use synthetic or grid based city layouts, and has been shown to synchronise the green phases of the main roads [30].

𝑟 𝑡 = −𝑃 𝑖 = −(𝑃 𝑢𝑝 ) − (𝑃 𝑑𝑜𝑤𝑛 ) Where(11)

7. Pressure Squared: Following on from pressure, we implemented pressure squared to test if penalising actions which lead to increased pressure is an effective approach to the reward function.

𝑟 𝑡 = −(𝑃 𝑖 ) 2(12)

8. Queue: This reward function is trivial to calculate and implement in the real world, and is used in some VA implementations. In addition, it is one of the most common reward functions used in implementations, as seen in [18].

𝑟 𝑡 = −|𝑉 𝑡 |(13)

9. Queue Squared: This reward function further penalises the actions which lead to larger queue. This was included due to the multi-agent scenario, as reducing the amount of queuing cars could increase the predictability of the traffic flow outbound from an intersection.

𝑟 𝑡 = −(|𝑉 𝑡 |) 2(14)

10. Maximum Wait Aggregated Queue (MWAQ): In this reward function, we use the value for the maximum waiting time multiplied with the length of the queue to approximate the worst case aggregate time waited for all the cars. This approach is a modification of the approach used by Ma et al. [14].

𝑟 𝑡 = −(max {𝑣∈𝑉 𝑡 } 𝜏 𝑡 * ∑ 𝑛∈𝑁 𝑞 𝑡 )(15)

11. Neighbourhood Adjusted Maximum Wait (NAMW): In this approach, we include basic information (number of vehicles) from a neighbouring intersection, as demonstrated in [31]. This may pose some implementational problems in the real world use due to the changing nature of traffic networks. However, this information is collectable via the most common type of sensor used in UK roads, induction loop sensors, which are low-cost and effective. In addition, it is possible to retrofit these sensors into existing infrastructure [32].

𝑟 𝑡 = −(max {𝑣∈𝑉 𝑡 } 𝜏 𝑣 + 𝛾 max {𝑣∈𝑉 𝑡𝑛 } 𝜏 𝑣 )

Where 𝑉 𝑡𝑛 is the vehicles at neighbour intersections (16) In the definition of NAMW, we include an additional discount factor 𝛾. This value is applied to the information from the neighbouring intersections, to ensure that the component of this function which has the greatest impact on the overall value is the component from the agent in question. 12. MOVA (referred to as VA in our results): As a benchmark, we used an implementation of one of the most commonly found TSC algorithms in the UK, MOVA [1].

Experimental Setup

We used the RESCO benchmarking environment as introduced by Ault et al. [6], based on the Simulation of Urban MObility (SUMO) simulator. Included in RESCO is the Ingolstadt environment [33], a demand-calibrated scenario for SUMO. The traffic network and traffic demand were set up as described in the Ingolstadt scenario [33].

We chose to use Deep Q-Learning for all of our agents as Deep Q-Learning is commonly used within the literature [18,34]. Furthermore, it was found by Genders et al. that the agent is not sensitive to the state representation [35], and so in our experiments we chose to use the state representation provided by [20]. This state definition at an intersection includes number of vehicles in each incoming lane, the speed of the incoming vehicles, the queue length and the total waiting time of the vehicles at that intersection. The DQN used was implemented in PyTorch, and included a convolutional layer, followed by two fully connected layers of 32 neurons. The parameters for the DQN were set as in [20].

Each reward function was repeated with 𝑛 = 20, with the total waiting time calculated for each run, and the average of this cumulative waiting time was used to evaluate the functions. Moreover, in our initial experiments, we found that the traffic scenario did not include enough vehicles to saturate the road network, and definitively test the reward functions. In order to resolve this, we chose to modify the traffic scale option within SUMO. This option, which is set to 1 by default, proportionally increases the traffic by that percentage. We set it to 1.5, meaning that each car in the network had a 50% chance of being duplicated. We chose this instead of generating random data as it would still maintain the flow of traffic which is seen in the Ingolstadt dataset. We chose a scale of 1.5 as it was a compromise, due to a quirk in how SUMO processed uncompleted journeys at the end of the simulation. If cars do not arrive at their destination by the simulation end time, they are not included in the output data, leading to misleading information as worse-performing agents outperform those which can (despite long delays) allow a greater throughput of vehicles.

Results

Figures 1 and 2 contain box plots of our results for traffic scale factors of 1 and 1.5, respectively. Tables 1 and 2 contains the tabular waiting time results for the traffic scale factor of 1 and 1.5 Our initial run with the default traffic flow found that pressure (pressure and pressure squared) and average speed (average speed and average speed normalised) based methods were the only methods to not outperform the MOVA/VA benchmark when the traffic scale factor was set to 1 and the traffic did not saturate (or near-saturate) the network. Whilst no conclusions can be made between the RL algorithms in this scenario, 7 of the reward functions used outperformed the benchmark, showing that there is significant potential in the use of RL for TSC. Furthermore, we note that the average speed and ASN reward functions performed significantly worse than in [17] when used in a multi-agent scenario. We speculate that this is caused by the problem of non-stationarity as agents could struggle to differentiate what is causing their low reward results when a signal upstream is essentially controlling the flow of traffic into that junction. Furthermore, junctions which see few vehicles passing through are likely to be impacted more by the changing policy of upstream intersections, a factor which could penalise the average speed functions moreso.

In addition, the pressure based reward functions did not outperform the benchmark either. We hypothesise that this is in part caused by the structure of the road layout, and the type of road layout which was used to develop these algorithms. Whilst these algorithms may perform well in arterial road layouts [4] and grid based layouts, they may struggle when faced with other road networks. It's important to note that this kind of road layout is rare in Europe, which is where the data originates from.

Once the traffic scale was increased to 1.5, a greater divergence is seen between the functions used. We see the trend of average speed and pressure being outperformed by the baseline. It is also important to note the reduced variance in the VA baseline. In real world deployments, this may be important as it increases the predictability of the algorithm.

Whilst the MWAQ was only slightly better than the other algorithms at the higher traffic scale, the reduced variance in the runs mean that in almost every run, it outperformed the other algorithms. We believe that this improvement is due to the fact that when only queue metrics are used, there is chance that the priority will almost certainly be given to the direction which has the greatest flow of traffic, leaving some cars to wait for a significant amount of time as they are travelling perpendicular to the flow of traffic. When the maximum wait time is included, the agent is less likely to prioritise the flow of traffic as often. Results from Ingolstadt dataset with a traffic scale factor of 1.5. All measurements are in seconds, and sorted by the mean.

Conclusion

In this paper, we discuss the non-stationary problem and how it may impact the use of RL for the TSC problem. We evaluate 11 different reward functions, including some of the more commonly used examples, and compare them to a benchmark of a real world function through simulations on a calibrated dataset from Ingolstadt. We believe that one potential avenue of further work is to conduct these experiments on a larger scenario, which would allow for further validation of the optimal reward function for use in the TSC problem. There may also be benefits to an ensemble approach to the TSC, where multiple agents with different reward functions are used to come to a conclusion on the optimal decision. Moreover, an approach which could be explored is to employ pretraining on new agents, training each agent on an individual intersection with the same number of lanes as the one they will control before being implemented in the network with other agents. Whilst this may increase the time required to train an agent, it may allow all agents to converge on a solution sooner.

Additionally, there could be a focus on the environmental impacts of using one reward function over another. For example, HGVs emit significantly more emissions when they accelerate compared to private vehicles. Therefore, an algorithm which does not differentiate between these types of vehicles will not prioritise this (or the impact on traffic once the HGV slows down, and the corresponding environmental impact of this), and therefore cause more damage to the environment.