Autonomous intersection management (AIM) poses significant challenges due to the intricate nature of real-world trafic scenarios and the need for a highly expensive centralised server in charge of simultaneously controlling all the vehicles. This study addresses such issues by proposing a novel distributed approach to AIM utilizing multi-agent reinforcement learning (MARL). We show that by leveraging the 3D surround view technology for advanced assistance systems, autonomous vehicles can accurately navigate intersection scenarios without needing any centralised controller. The contributions of this paper thus include a MARL-based algorithm for the autonomous management of a 4-way intersection and also the introduction of a new strategy called prioritised scenario replay for improved training eficacy. We validate our approach as an innovative alternative to conventional centralised AIM techniques, ensuring the full reproducibility of our results. Specifically, experiments conducted in virtual environments using the SMARTS platform highlight its superiority over benchmarks across various metrics.
CEUR ceur-ws.org
(G. A. Susto) CEUR Workshop Proceedings
A vast literature exists on AIM. The research in this field spans multiple fronts, each
leveraging distinct methodologies to address the challenges of optimizing trafic flow and ensuring
safety in dynamic urban environments. By employing reinforcement learning (RL), AIM systems
can efectively learn and adapt intersection control strategies in response to changing trafic
conditions [
Despite the advancements in AIM techniques, their implementation still faces important
challenges. One of the main obstacles is represented by the need for an expensive centralised
server which has to be positioned in the proximity of the intersection, in order to simultaneously
control all the vehicles. Moreover, the vehicles should continuously send their local information
to this centralised controller, which will then gather and elaborate the data coming from all the
road users, before sending back to each vehicle a velocity or acceleration command. Given the
complexity and high demands of this technological framework, the integration of AIM devices
into existing transportation infrastructures still requires many years of extensive research and
testing. In this direction, we devise an alternative distributed approach based on the 3D surround
view technology for advanced assistance systems [
1The Python code of our work can be found at https://github.com/mcederle99/MAD4QN-PS. The authors want to stress the fact that reproducibility represents a crucial issue within this field of research.
The remainder of this paper unfolds as follows. The preliminaries for this study are yielded in Section 2; whereas, Section 3 provides the core of our contribution, namely the multi-agent decentralised dueling double deep q-networks algorithm with prioritised scenario replay (MAD4QNPS). This innovative method is then tested and validated through several virtual experiments, as illustrated in Section 4. Finally, Section 5 draws the conclusions for the present investigation, proposing future developments.
Notation: The sets of natural and positive (zero included) real numbers are denoted by ℕ and ℝ0+, respectively. Given a random variable (r.v.) , its probability mass function is denoted by [ = ] , whereas [ = | = ]
indicates the probability mass function of conditioned to the observation of a r.v. . Moreover, the expected value of a r.v. is denoted by [ ] .
RL is a machine learning paradigm in which an agent learns to solve a task by iteratively
interacting with its environment. Solving the task means maximising the cumulative rewards obtained
over time. A generic RL problem is formalised by the concept of Markov decision process (MDP)
[
performed by the is used to measure the quality of each transition, while ∈ [0, 1) denotes a discount factor, used to compute the cumulative reward at time , i.e. the return = ∑=0 ∞
++1 . The agent decides which action to take at each iteration exploiting its policy, a function that maps any state to the probability of selecting each possible action: (|) = [
= ∣ = ], ∀ ∈ .
Solving a RL problem means finding an optimal policy ∗. One criterion that is usually adopted to find ∗ consists in the maximization of the state-action value function (, ) , i.e. the expected return starting from state ∈ , taking action ∈ , and thereafter following policy : (, ) = [ ∣ = , = ] .
Consequently, given the state-action value function, the optimal policy is defined as ∗ = arg max (, ) . There is therefore an inherent relation between ∗ and the optimal stateaction value function.
Finally, two other important quantities which will be used in the proceeding of this article are the state value function and the advantage function. The former is defined as the expected return starting from state ∈ and then following policy :
() = [ ∣ = ] .
(, ) = (, ) − ().
The latter instead is used to give a relative measure of importance to each action for a particular state, and it is defined starting from (, ) and () : (1) (2) (3) (4)
To compute the optimal state-action value function we could theoretically exploit the recursive
Bellman Optimality Equation [
′
however due to the curse of dimensionality and the need for perfect statistical information to
compute the closed-form solution, it is necessary to resort to iterative learning strategies even
to solve simple RL problems. The most common algorithm used in literature is Q-Learning [
MARL expands upon traditional RL by incorporating multiple agents, each making decisions in
an environment where their actions influence both the immediate rewards and the observations
of other agents. In its most general definition, a MARL problem is formalised as a partially
observable stochastic game (POSG), in which each agent has its own action space and reward
function. Moreover, the partial observability derives from the fact that the agents do not perceive
the global state, but just local observations, which carry incomplete information about the
environment [
MARL algorithms can be categorised depending on the type of information available to the
agents during training and execution: in centralised training and execution (CTCE), the learning
of the policies as well as the policies themselves use some type of structure that is centrally
shared between the agents. On the other hand, in decentralised training and execution (DTDE),
the agents are fully independent and do not rely on centrally shared mechanisms. Finally,
the centralised training and decentralised execution paradigm (CTDE) is in between the first
two, exploiting centralised training to learn the policies, while the execution of the policies
themselves is designed to be decentralised [
In this section we present our novel method based on MARL, called Multi-Agent Decentralised
Dueling Double Deep Q-Networks with Prioritized Scenario replay (MAD4QN-PS). We begin
by detailing how the system is modelled, and then we describe the original learning procedure
that we implement in order to train agents through self-play [
The environment in which the agents live consists of a 4-way 1-lane intersection, with three diferent turning intentions available to each vehicle.
Recalling Section 2.3, we formalize the problem as a POSG, which can be seen as a multi-agent extension to MDPs. For this reason, we shall define for each agent the observation space, the action space and the reward function.
The information retrieved by each vehicle at every time step consists of a local RGB bird-eye view image with the vehicle at the center. As already discussed in Section 1, this type of data is already recoverable from sensors with modern technology, thus making such a configuration extremely interesting from an application point of view. Moreover, the final observation passed to the agent is represented by a stack of ∈ ℕ consecutive frames, thus allowing the algorithm to capture temporal dependencies and understand how the environment is changing over time.
The action space of each agent instead is discrete and it contains ∈ ℕ velocity commands.
This choice has been made because the purpose of our algorithm is not to learn the basic
skills required for driving, such as keeping the lane and following a trajectory, but it is instead
choosing how to behave in trafic conditions and when to interact with other vehicles present
in the environment. Moreover, a similar high-level perspective has also been implemented in
other works, related to the centralised AIM paradigm [
Finally, for what concerns the reward function, we need to take into consideration the fact that each agent is trying to solve a multi-objective problem. Indeed the main goal of each vehicle is crossing the intersection and reaching the end of the scenario. In the meantime, a vehicle is also required not to collide with the others, by travelling as smoothly as possible. In order to fulfill all these objectives we design a reward signal composed by diferent terms: = ⎧+ ⎪− ⎨−10 ⋅ ⎪ ⎩+10 ⋅ if > 0, if vehicle not moving, if a collision occurs, if scenario completed, (9) (10) (11) where ∈ ℝ 0+ is the distance travelled in meters from the previous time step and ∈ ℕ is a hyperparameter used to weight the importance of the last three components of the reward function with respect to the first one.
The starting point for our learning strategy is the algorithm Deep Q-Networks, already presented in Section 2.2. This algorithm is then slightly modified by considering the Double DQN scheme and also the Dueling architecture, which will be briefly introduced in the sequel.
The idea of Double DQN [
Dueling DQN [
The final algorithm used for training is therefore a Multi-Agent version of Dueling Double DQN known as D3QN, with linearly-annealed -greedy policy for all the agents. In order to allow for decentralised execution while developing at the same time a smart training strategy, we consider an intermediate approach between the DTDE and the CTDE paradigms. In particular, we initialize and train three diferent D3QN agents, one for each turning intention, i.e. left , straight and right. In this way each vehicle can select which model to use at the beginning of its path, according just to its own turning intention.
This approach is extremely sample-eficient, because we keep the number of network parameters constant, regardless of the number of vehicles considered. Moreover, these shared parameters are optimised through the experiences generated by all the vehicles, leading to a more diverse set of trajectories for training. Indeed, each of the three models has its own replay bufer, which contains transitions shared from all the vehicles with the corresponding turning intention. The crucial insight that makes our strategy efective is the fact that the observations gathered from each vehicle are invariant with respect to the road in which the vehicle itself is positioned. This parameter and experience sharing approach renders the training procedure of the algorithm somehow centralised because trajectories coming from diferent vehicles are used to train the three D3QN agents. However, we remark that, once the models have been trained, the execution phase is completely decentralised, since each vehicle locally stores the three diferent models. Then, at the beginning of the scenario, each CAV selects the model to use based only on its own turning intention.
The agents are trained for a fixed number of iterations ∈ ℕ , keeping the intersection busy in order to obtain meaningful transitions to learn from. In particular, at each episode we consider the most complicated situation in which there are four vehicles simultaneously crossing the intersection, one for each road and with random turning intention.
Every ∈ ℕ time steps we pause training and run an evaluation phase. During this period,
the agents use a greedy policy to face all the possible scenarios described above. When the
evaluation is completed, we use the inverse of the returns from all the scenarios to build a
probability distribution, and in the following training window we sample the diferent scenarios
according to such a distribution. In this way we allow the agents to learn more from the
most complicated situations. We name this original training strategy prioritised scenario replay
because of its conceptual similarity with the prioritised experience replay scheme [
In order to train and evaluate our algorithm we need a suitable simulation environment. For this
project we have chosen the platform SMARTS [
To develop our code Python 3.8 was employed along with the version 1.4 of the Deep Learning
library PyTorch [
end if end while 39: end while Algorithm 1 MAD4QN-PS 1: Initialize three state-action value networks with random parameters , = 1, 2, 3 2: Initialize three target state-action value networks ̄ with parameters ̄ = , = 1, 2, 3 3: Initialize three replay bufers , = 1, 2, 3 4: Setup initial , decay factor , evaluation period , target update period , discount factor 5: Uniformly initialize the scenarios probability distribution 6: max_episode_steps ← 7: ← 0 8: while <
do episode_terminated ← False Randomly reset the environment episode_steps ← 0 while not episode_terminated do
Update target network parameters ̄ = for each agent = 1, 2, 3
Run evaluation phase and update the scenarios probability distribution as described episode_steps ← episode_steps + 1 end for ← − ← + 1 if % == 0 then end if if % == 0 then
in Section 3.3 end if ←
number of vehicles currently present in the simulation Assign each vehicle to one of the three agents, based on its turning intention for all vehicles in 1, ..., do
With probability select a random action Otherwise ← arg max ( , ; ), where depends on the turning intention of end for
Store each transition in the corresponding replay bufer , = 1, 2, 3 for all agents = 1, 2, 3 do
Sample random batch of transitions ℬ from replay bufer Update parameters by minimising the loss function:
1 |ℬ | ∈ℬ ℒ ( ) = ∑ [( + ̄ ( ′, arg max ( ′, ′;
); ̄ ) − ( , ; ))2] ′ if a collision occurred or == 0 or episode_steps == max_episode_steps then episode_terminated ← True The simulation step has been fixed to 100 . Regarding the observation of each vehicle, we have stacked = 3 consecutive frames, each consisting of a RGB image of dimensions 48 × 48 pixels. Whereas, the action space contains = 2 possible velocity commands2, namely 0 and 15 / . The chosen velocity references are then fed at each iteration to a speed controller, in charge of driving the vehicle until the subsequent time step. For what concerns the reward function (9), we have fixed its hyperparameter to = 1 . Regarding the architecture of the NN used to approximate the state-action value function, we have considered a convolutional neural network (CNN), whose structural details are summarised in Table 1. Finally, the training hyperparameters of Algorithm 1 are reported in Table 2.
In order to assess the quality of our algorithm, we benchmark it versus the following baselines 3:
• Random policy (RP) for all the vehicles, which helps confirm whether our algorithm is
efectively learning meaningful patterns, as it demonstrates its ability to outperform
random actions, which lack any deliberate learning process.
• Three symmetric (N/S & W/E) fixed-time trafic lights
(FTTL), considering two cycle
lengths already analyzed in [
3The baselines simulations with trafic lights have been performed exploiting Flow [ 36], another platform used to interface with SUMO, which easily allows for the definition and control of trafic lights.
• Two symmetric (N/S & W/E) actuated trafic lights (ATL) [
To assess the quality of MAD4QN-PS we consider four metrics, namely the travel time, the waiting time, the average speed and the collision rate. We remark that we have accounted for vehicle-centered metrics because of the decentralised nature of our algorithm. However, it is evident that by optimizing the performance of each single road user we also implicitly improve the quality of the whole intersection. The robustness of our method is ensured by performing training ten times across diferent seeds, and then considering all the diferent trained models while evaluating our strategy. In particular, each model has been tested by running a cycle of the evaluation phase presented in Section 3.3, considering 600 veh/hour as flow rate of vehicles coming through the intersection. Then, the obtained results from the diferent models have been averaged out. Moreover, to ensure a fair comparison and analysis of the results, the same evaluation setup has been adopted for all the baselines introduced above. Finally, it is worth noticing that the inference time of the networks at evaluation phase is at most 1 , thus allowing for real-time control, given that the simulation step has been fixed to 100 , as discussed at the beginning of this section.
Starting from Figure 1a, we can observe the average travel time and waiting time of a generic vehicle for all the methods. The former is defined as the overall time that the vehicle spends inside the environment, while the latter is defined as the fraction of the travel time in which the vehicle is moving with velocity less or equal to 0.1 / , i.e. when it is stopped or almost stopped. We clearly see that our method strongly outperforms all the trafic lights configurations. This is mainly due to the fact that, when using trafic lights, a fraction of the vehicles is forced to stop as soon as the corresponding light becomes red. Conversely, the trained MAD4QN-PS agents are able to smoothly handle the interaction among multiple vehicles, allowing them to avoid stopping unless it is strictly necessary. Figure 1b instead displays the average speed of each vehicle. The results shown in this histogram are clearly related to those in Figure 1a; indeed, also in this case we can see that our method outperforms trafic lights control schemes. This occurs since the vehicles almost never stop, thus keeping a smoother velocity profile throughout all the duration of the simulation.
Lastly, we are left with the analysis of the random policy baseline, as we need to look at all
the three plots to fully understand its behaviour. If we just look at Figures 1a and 1b we could
argue that the random policy performance is similar to that of MAD4QN-PS. This hypothesis is
however disproved by Figure 1c, where the average collision rate for each vehicle is illustrated.
The extremely high collision percentage obtained by the random policy explains why each
vehicle on average spends a small amount of time with high velocity into the environment.
Indeed, the simulation is terminated as soon as a vehicle crashes. MAD4QN-PS, instead, achieves
an extremely low collision rate. In addition, the fact that such a collision rate is non-zero is
expected and also observed in other works exploiting RL-based techniques [
A short video showing the performances of MAD4QN-PS can be found at this link.
In this study, we consider a distributed approach to face the AIM paradigm. In particular, we propose a novel algorithm which exploits MARL through self-play and an original learning strategy, named prioritised scenario replay, to train three diferent intersection crossing agents. The derived models are stored inside CAVs, that are then able to complete their paths by choosing the model corresponding to their own turning intention while relying just on local observations. Our algorithm represents a feasible and realistic alternative to the centralised AIM concept, that is still expected to require years of technological advancement to be implementable in a realworld scenario. In addition, simulation experiments demonstrates the superior performances of our method w.r.t. classic intersection control schemes, such as static and actuated trafic lights, in terms of travel time, waiting time and average speed for each vehicle.
In future works, we aim to explore diferent directions for advancements. In particular, one of the main objectives is to also consider human driven vehicles inside the environment and extend our approach to this field of research (see, e.g. the initial efort made in [ 38]). In this case, the most challenging issue is indeed represented by the synchronization of trafic lights accounting for the presence of human driven vehicles. Moreover, given the decentralised nature of the proposed method, we expect to render our algorithm more robust without dramatically
Average travel time per vehicle Average waiting time per vehicle 8 7 /[]56 s m d e4 e p 3 S 2 1 0 FTTL1 FTTL2 FTTLOPT ATL1 ATL2 RP MAD4QN-PS
0 FTTL1 FTTL2 FTTLOPT ATL1 ATL2 RP MAD4QN-PS (a) Average travel and waiting time per vehicle (b) Average speed per vehicle 80 []%60 e g a t en40 c r e P 20 0
RP
MAD4QN-PS (c) Average collision rate change it. Conversely, significant redesign would be necessary for a centralised AIM approach. Furthermore, we envisage to test more complicated scenarios, both in terms of dimension and layout, again to improve the robustness of our algorithm. Finally, we intend to implement our algorithm in a scaled real-world scenario with miniature vehicles [39], to practically demonstrate the applicability of our method.
This study was carried out within the MOST (the Italian National Center for Sustainable Mobility), Spoke 8: Mobility as a Service and Innovative Services, and received funding from NextGenerationEU (Italian PNRR – CN00000023 - D.D. 1033 17/06/2022 - CUP C93C22002750006).
L. Antiga, A. Lerer, Automatic diferentiation in pytorch (2017). [35] G. Hinton, N. Srivastava, K. Swersky, Lecture 6a overview of mini–batch gradient descent, Coursera Lecture slides https://class. coursera. org/neuralnets-2012-001/lecture,[Online (2012). [36] C. Wu, A. R. Kreidieh, K. Parvate, E. Vinitsky, A. M. Bayen, Flow: A modular learning framework for mixed autonomy trafic, IEEE Transactions on Robotics 38 (2021) 1270–1286. [37] F. V. Webster, Trafic signal settings, Technical Report, 1958. [38] Z. Yan, C. Wu, Reinforcement learning for mixed autonomy intersections, in: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), IEEE, 2021, pp. 2089–2094. [39] L. Paull, J. Tani, H. Ahn, J. Alonso-Mora, L. Carlone, M. Cap, Y. F. Chen, C. Choi, J. Dusek, Y. Fang, et al., Duckietown: an open, inexpensive and flexible platform for autonomy education and research, in: 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 1497–1504.