Introduction

As a result of the increasing trend in private vehicle ownership, there are over a billion vehicles in the world's motor fleet currently, and this is expected to continue growing in the near future [1]. This trend is likely to cause increased congestion and road accidents. According to a report from the World Economic Forum (2018), road congestion cost 87 billion dollars to the US economy in 2018 due to loss of productivity [2]. Furthermore, a European Union (EU) report states that around 78% of road crashes are considered to be a result of human errors [3]. To minimise congestion and improve traffic safety, Autonomous Vehicles (AVs) are considered one of the main interventions in Intelligent Transportation Systems (ITS) [4].

AV technologies are evolving with the developments in communication technologies and Artificial Intelligence (AI). Connected Autonomous Vehicles (CAVs) are leveraging recent advancements in vehicular communication (V2X) technologies to make collaborated manoeuvres to improve traffic safety and efficiency. AI has been a popular option to solve some of the complex problems in AV technologies, such as localisation, mapping, perception, route planning, and motion control [5]. For CAV motion controllers specifically, our previous work shows that Multi-Agent Reinforcement Learning (MARL) is a popular choice [6]. Lane changing is one of the complex problems in motion control, as improper lane change may cause a collision that could damage the costly components in AVs or even cause loss of lives. Many forms of MARL using Deep Q-Networks (DQNs) [7,8,9], and Actor-Critic Networks (ACN) [10,11,12] has been applied for designing lane change controllers. Among them, MARL-CAV [12] is an open-sourced state-of-the-art MARL lane change controller designed for CAVs [13]. The MARL-CAV significantly improves traffic efficiency and safety. This approach, however, uses a predication-based priority assignment to avoid collisions and encourage safe behaviour, and therefore safety is not guaranteed, which limits its applicability.

Our previous work identifies that Control Barrier Functions (CBFs) are suitable for ensuring the safety of CAV lane change controllers [13]. CBFs have been recently applied to ensure the safe operation of Reinforcement Learning (RL) based single-agent AV controllers [14]. This CBF implementation demonstrates a longitudinal safety constraint in a simple scenario. The CBF can be formulated by considering dynamic safety constraints relative to the surrounding vehicles [15]. This single-agent CBF, however, assumes that other agents make worst-case decisions. Such a safety constraint results in conservative behaviour, negatively affecting traffic efficiency.

Overall, CAV lane change controllers can be designed using MARL to improve traffic efficiency, but they do not ensure safety. This design aims to integrate the CBF safety constraints with the MARL-based lane change controller [12] to ensure safety by considering multi-agent vehicular dynamics to design safety constraints. The main contributions of this work are:

• The architecture to integrate CBF constraints to the high-level MARL-based lane change controllers (Section 3). • The structure for defining the dynamics of multi-agent interaction between CAVs (Section 4.2).

• The specifications and formulations of the CBF constraints to ensure the safety of CAVs (Section 4.1 and Section 4.3).

Background

The background details related to AV control hierarchy, vehicle dynamics, RL, and CBFs are provided in this section. First, the scope of this research is explained based on control hierarchies. Next, the kinematic bicycle model is explained, and the assumptions related to vehicle dynamics are outlined. Then, notations used for RL formulations are discussed. Finally, a general form of a CBF is defined along with an optimisation problem for evaluating the safe control inputs.

Hierarchy of control layers

The control decisions of AVs can be separated into four hierarchical levels such as route planning, behavioural layer, motion planning, and local feedback control [16]. The route planning layer first identifies a feasible route to the destination provided by the user using the road network information. The route generated from this layer consists of a sequence of waypoints. While moving along these waypoints, the behavioural layer makes high-level driving decisions such as following a lane, performing a lane change, negotiating at the intersection, or moving in an unstructured environment. The motion planning layer generates reference control actions, such as acceleration and steering, to execute a specific manoeuvre from the high-level decision. In the last layer, a local feedback controller performs necessary actuation, such as steering, throttling, and braking, to follow the control references. The lane change controller can be developed by engineering high-level behavioural and low-level motion planning layers. Specifically, a behavioural layer can be designed to make discrete decisions to change lanes or follow the current lane. Based on this decision, the motion planning layer can identify the control references to execute the desired driving manoeuvre. Therefore, this article mainly focuses on designing these two layers in the AV control hierarchy.

Vehicle dynamics

In this article, the kinematic bicycle model is considered to define vehicle dynamics. This model considers the two wheels in the front as one wheel and the same for the back wheels, as illustrated in Figure 1. The distance between the front and back wheels is denoted as 𝑉 𝑙 . The vehicle's position is defined using 𝑥 and 𝑦, longitudinal and lateral coordinates along the road. The vehicle's velocity (𝑣) is controlled by adjusting the acceleration input (𝑢 1 ), and the steering angle (𝛿) is controlled by adjusting the steering velocity (𝑢 2 ). The steering velocity represents the rate of change in steering angle with time [17]. The steering angle is considered the same as the angle of the front wheels with respect to the current heading of the vehicle (𝜓). The equations for the kinematic bicycle model that assumes the centre of gravity (𝐺) on the axle with equal distance from the front and back wheels can be written as [18],

ẋ = 𝑣 𝑥 , ẏ = 𝑣 𝑦 , v 𝑥 = 𝑢 1 cos (𝜓 + 𝛽), v 𝑦 = 𝑢 1 sin (𝜓 + 𝛽), ψ = 𝑣 𝑉 𝑙 sin 𝛽, δ = 𝑢 2(1)

where 𝛽 is a slip angle at the centre of gravity:

𝛽 = tan −1 ( 1 2 tan 𝛿)(2)

Reinforcement learning

RL is a computational approach for learning a sequence of actions to achieve a specific goal. The RL problem is formulated using a Markov Decision Process (MDP) defined by the tuple (𝒮 , 𝒜 , 𝒫 , ℛ, 𝛾 ). 𝒮 is the state space, the set of state variables that an agent can observe. An agent observes a state 𝑠 𝑡 ∈ 𝒮 at a time step 𝑡. 𝒜 is the action space consisting of the set of actions that an agent can perform. At a time step 𝑡, an agent performs an action 𝑎 𝑡 ∈ 𝒜. 𝒫 ∶ 𝒮 × 𝒜 × 𝒮 → [0, 1] is the state transition function that defines the likelihood of changes in the state observed from the environment based on an action 𝑎 𝑡 ∈ 𝒜. ℛ ∶ 𝒮 × 𝒜 × 𝒮 → ℝ is a reward function that defines the agent's goal. At a time step 𝑡, the agent receives a reward 𝑅 𝑡 , which is a real number calculated for a transition from the previous state to the current state through an action. The reward function formulation plays a vital role in defining agents' behaviour in the system. 𝛾 ∈ (0, 1] is a discount factor used to define the discounted reward 𝐺 𝑡 ,

𝐺 𝑡 = ∞ ∑ 𝑘=𝑡+1 𝛾 𝑘 𝑅 𝑘

The discounted reward can provide a measure to choose an action that has higher probability of getting better rewards in the future. Using the discounted reward, a state-action value function, also known as the Q-value, can be derived for a policy. A policy 𝜋 is a mapping from states to the probability of selecting possible actions. The Q-function under policy 𝜋 provides an expected future reward by choosing action 𝑎 𝑡 from state 𝑠 𝑡 . It can be defined as,

𝑄 𝜋 (𝑠 𝑡 , 𝑎 𝑡 ) = 𝔼[𝐺 𝑡 |𝑠 𝑡 , 𝑎 𝑡 ]

For simple problems with a small number of possible states and actions, Q-values can be calculated based on the transition probability 𝒫 using the following equation:

𝑄 𝜋 (𝑠 𝑡 , 𝑎 𝑡 ) = ∑ 𝑠 𝑡+1 𝒫 (𝑠 𝑡 , 𝑎 𝑡 , 𝑠 𝑡+1 )[𝑅 𝑡 + 𝛾 max 𝑎 𝑡+1 𝑄 * (𝑠 𝑡+1 , 𝑎 𝑡+1 )]

For complex tasks with a large state and action space, such as autonomous driving, it is often very difficult or impossible to model the transition probability 𝒫. Therefore, approximation methods are usually used to find a policy to achieve higher rewards in such tasks. These approximations can be implemented using deep neural networks [19]. Deep RL (DRL) approximation algorithms achieved impressive results in playing Atari games [20]. Some of the recent RL approximation algorithms include Deep Q-Networks (DQN) and policy gradient methods, such as Actor-Critic Networks (ACNs). The open-source ACN algorithms such as PPO [21] and ACKTR [22] have been developed and published on repositories like StableBaselines-3 [23] and OpenAI baselines [24]. These algorithms are applied to solve optimisation problems in various research areas, including manufacturing, robotics, large language models, and autonomous vehicles.

The RL algorithms extended to MAS considering various forms of learning and control components are known as MARL [25]. The learning components learn an approximate optimal policy, and the control components execute that policy. These components are integrated into an agent in single-agent tasks such as a robot cleaning the house. Many real-world tasks, however, can be considered to be MAS, as multiple agents may need to work together in the same environment. For example, systems such as multiplayer online games, cooperative robots in factories, traffic control systems, and CAVs can be considered as MAS [26]. However, MARL applications are limited to non-safety-critical tasks, as they can not ensure safety because of the blackbox property [27]. To overcome this limitation, CBF safety constraints can be used.

Control barrier functions

Consider a discrete time nonlinear control system defined by the following transition dynamics ṡ = 𝑓 (𝑠 𝑡 ) + 𝑔(𝑠 𝑡 )𝑢 𝑡 (3) where change in state variables ṡ per unit time is defined using unactuated dynamics 𝑓 ∶ 𝑆 → 𝑆, and actuated dynamics 𝑔 ∶ 𝑆 → ℝ 𝑛,𝑚 , 𝑛 and 𝑚 are number of variables in the state space 𝑆 and action space 𝑈 respectively, 𝑠 𝑡 ∈ 𝑆 is the system state, and 𝑢 𝑡 ∈ 𝑈 is the control action at time step 𝑡. The 𝑓 and 𝑔 are defined based on known system dynamics and they are locally Lipschitz continuous, in other words, continuous functions limited by a maximum rate of change. For example, the kinematic bicycle model defined in equation ( 1) can be defined as a discrete time nonlinear control system (3) as follows,

ṡ = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 𝑣 𝑥 𝑣 𝑦 0 0 𝑣 𝑉 𝑙 sin 𝛽 0 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ + ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 0 0 0 0 cos (𝜓 + 𝛽) 0 sin (𝜓 + 𝛽) 0 0 0 0 1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ [ 𝑢 1 𝑢 2 ](4)

Consider a safe set 𝐶 defined as the super-level set of the continuously differentiable function

ℎ ∶ 𝑆 → ℝ 𝐶 ∶ {𝑠 𝑡 ∈ 𝑆 ∶ ℎ(𝑠 𝑡 ) ≥ 0}(5)

To ensure the safety of the control system (3), the safe set 𝐶 must be forward invariant. When 𝐶 is forward invariant, safe actions can be defined for each state 𝑠 𝑡 ∈ 𝐶 such that the system continues to stay in 𝐶. The safe set 𝐶 is considered invariant if the function ℎ is a Control Barrier Function (CBF) such that there exists 𝜂 ∈ [0, 1] for all 𝑠 𝑡 ∈ 𝐶 satisfying the following equation ( 6)

sup 𝑢 𝑡 ∈𝑈 [ℎ (𝑓 (𝑠 𝑡 ) + 𝑔(𝑠 𝑡 )𝑢 𝑡 ) + (𝜂 − 1)ℎ(𝑠 𝑡 )] ≥ 0 (6)

where 𝜂 defines the magnitude at which the system is pushed within the safe set 𝐶 [14]. Using smaller values of 𝜂 can enforce the constraints strictly, whereas higher values can relax the constraints. Therefore, 𝜂 represents how strongly the barrier function pushes the states inwards within 𝐶. The existence of a CBF implies that for all 𝑠 𝑡 ∈ 𝐶, there exist 𝑢 𝑡 such that 𝐶 is forward invariant [28]. Therefore, the goal is to find a minimal safe action 𝑢 cbf 𝑡 that satisfies (6) to ensure the safety of a control system (3). Let us consider the affine barrier function of the form ℎ(𝑠 𝑡 ) = 𝑝 T 𝑠 𝑡 + 𝑞 (7) where 𝑝 ∈ ℝ 𝑛 and 𝑞 ∈ ℝ are the parameters used to define a safety constraint ℎ on the state 𝑠 𝑡 . Combining the affine barrier function with the condition ( 6), the following constraint can be defined for the control action 𝑢 𝑡 ,

−𝑝 T 𝑔(𝑠 𝑡 )𝑢 𝑡 ≤ 𝑝 T 𝑓 (𝑠 𝑡 ) + 𝑝 T (𝜂 − 1)𝑠 𝑡 + 𝜂𝑞(8)

To consider multiple safety constraints defined using CBFs, 𝐶 can be considered as the intersecting half spaces defined by 𝑘 affine barrier functions [15]. The affine constraint on 𝑢 𝑡 can be defined by stacking all the constraints.

𝐴𝑢 𝑡 ≤ 𝑏, where, 𝐴 = [𝑎 1 , 𝑎 2 , ..., 𝑎 𝑘 ], with 𝑎 𝑖 = −𝑝 T 𝑖 𝑔(𝑠 𝑡 ) 𝑏 = [𝑏 1 , 𝑏 2 , ..., 𝑏 𝑘 ], with 𝑏 𝑖 = 𝑝 T 𝑖 𝑓 (𝑠 𝑡 ) + 𝑝 T 𝑖 (𝜂 − 1)𝑠 𝑡 + 𝜂𝑞 𝑖(9)

This constraint can be used to reformulate the CBF given by ( 6) into the following optimisation problem

𝑢 𝑡 = arg min 𝑢 𝑡 ||𝑢 𝑡 || 2 s.t 𝐴𝑢 𝑡 ≤ 𝑏,(10)

which can be efficiently solved in each time step using quadratic program [29].

Integrating CBF with MARL for highway merging

In the AV control hierarchy (Section 2.1), the safety constraints can be integrated with the motion planning layer to ensure safe lane change manoeuvres [30]. The safety constraints act as a shield to override the control decisions from the motion planning layer to ensure that a vehicle stays in a safe state [31]. The architecture to integrate the MARL behavioural layer, motion planning layer, and safety constraints is presented in this section to develop a safe MARL lane change controller, illustrated in Figure 2.

MARL behavioural layer

A vehicle, referred to as the ego vehicle, makes behavioural decisions based on its state information, measured by onboard sensors such as LIDAR, RADAR, Camera, GPS, and IMU, as well as information about the states of the surrounding 𝒩 vehicles. The ego vehicle can decide whether to change lanes, follow the lane, speed up, or slow down [12]. Since the ego vehicle can observe vehicles within the range of vehicular communication (V2X), the previously defined MDP (in Section 2.3) can be extended as a Partially Observable MDP(POMDP) for this MARL application. Moreover, V2X is assumed to be a perfect communication interface without any delays or packet drops. The MARL formulation defined in this section is similar to the MARL-CAV formulation, [12], with minor changes in the state space and reward function.

The state space 𝒮 𝒾 of a vehicle 𝑖 consists of state variables including,

• 𝑥 : The longitudinal position of the vehicle.

• 𝑦 : The lateral position of the vehicle.

• 𝑣 𝑥 : The longitudinal velocity of the vehicle.

• 𝑣 𝑦 : The lateral velocity of the vehicle.

• 𝜓 : The vehicle heading with respect to the road.

These variables are observed with respect to a global coordinate system, while observed vehicle state variables are relative to the ego vehicle. As the ego vehicle observes states from 𝒩 surrounding vehicles, the overall multi-agent state space is defined as a Cartesian product of the individual states, 𝒮 = 𝒮 0 × 𝒮 1 × 𝒮 2 × ... × 𝒮 𝒩 . The 𝒩 = 5 is observed to achieve the best performance [12]. We have added the heading 𝜓 to the state variables considered by MARL-CAV to capture the lane change intentions of the CAVs, which ensures safety.

The action space 𝒜 is the same as defined in MARL-CAV, which consists of five discrete variables representing a specific behaviour, namely, right lane change, left lane change, follow lane, speed up, slow down. The behavioural layer chooses one of these high-level actions. The low-level controller explained in Section 3.2 further executes these decisions.

The reward function constitutes rewards for avoiding collision 𝑟 𝑐 , maintaining desirable speed 𝑟 𝑠 , maintaining desirable headway 𝑟 ℎ , and feedback from the CBF evaluation 𝑟 𝑓 along with an associated weight 𝑤 * for each reward component. These weights can be tuned to prioritise the CAV objectives. Therefore, the reward for a CAV 𝑖 at time 𝑡 is defined as

𝑟 𝑖,𝑡 = 𝑤 𝑐 𝑟 𝑐 + 𝑤 𝑠 𝑟 𝑠 + 𝑤 ℎ 𝑟 ℎ + 𝑤 𝑓 𝑟 𝑓

The feedback from the CBF evaluation, 𝑟 𝑓 , is an additional component added to the reward formulation used in MARL-CAV. This can reward the agent for staying in the safe state, which minimises the control overrides required from the CBF layer. Further, this reward encourages the agent to explore within the safe states [14]. Note that the reward 𝑟 𝑖,𝑡 is the reward associated with an individual agent. To achieve collaborative goals, MARL-CAV combines rewards from the surrounding agents to define a local reward as

𝑅 𝑖,𝑡 = 1 𝒩 𝒩 ∑ 𝑗=0 𝑟 𝑗,𝑡

The MARL-CAV is demonstrated with multiple RL algorithms, such as Multi-Agent extensions (MA*) of PPO [21], ACKTR [22], and DQN [20], namely MAPPO, MAACKTR, and MADQN. The multi-agent extension of these algorithms is inspired by the parameter sharing approach proposed in the Multi-Agent Actor Critic (MA2C) RL algorithm [32]. Among them, MAPPO performed best compared to other algorithms in Chen et al. 2023's MARL benchmark analysis [12]. Therefore, this work considers the MAPPO algorithm to train the high-level behavioural layer.

Motion planning layer

For the low-level motion planning layer, a Proportional-Integral-Derivative (PID) controller can be used to generate control actions, such as acceleration and steering velocity, to execute a behavioural command (defined in Section 3.1). Because of its simplicity, the PID controller can generate control actions in real time. Moreover, it does not require any pre-defined model. While it is possible to integrate the behavioural and motion planning layer using learning-based methods to design end-to-end controllers, they have been criticised for the difficulty in training policies to perform complex tasks [33]. Especially for autonomous driving tasks with dynamic surroundings, end-to-end controllers suffer from poor sample efficiency, resulting in high resource requirements [27]. Another option to integrate the high-level and low-level control layers with learning-based methods is to use hierarchical RL [34]. However, this approach is difficult to reproduce because of the complex training process. Other model-based approaches, such as Model Predictive Controller (MPC), require a model for generating low-level control actions [35]. Estimating such a model for generating CAV control actions in a complex scenario can be difficult. Therefore, the PID controller is a viable option for the low-level control layer along with the high-level MARL controller.

Since the high-level MARL controller is not guaranteed to make safe control decisions, the low-level controller can generate unsafe control actions. The low-level control action 𝑢 ll 𝑡 at time 𝑡 generated from the PID controller aims to execute the high-level behavioural decision. Therefore, the control action 𝑢 ll 𝑡 must be constrained to ensure safety.

Safety with CBF shield

Safety of a control system can be ensured by overriding the possibly unsafe lower-level control action 𝑢 ll 𝑡 with a correction 𝑢 cbf 𝑡 to comply with safety constraints defined using CBFs [14]. Therefore, the final control action 𝑢 𝑡 can be defined as

𝑢 𝑡 = 𝑢 ll 𝑡 + 𝑢 cbf 𝑡 (11)

With the updated definition for the action 𝑢 𝑡 (11), the constraints defined in ( 9) can be updated to modify the optimisation problem (10) as follows

𝑢 cbf 𝑡 = arg min 𝑢 cbf 𝑡 ||𝑢 cbf 𝑡 || 2 s.t 𝐴𝑢 cbf 𝑡 ≤ 𝑏 ll , where 𝑏 ll = [𝑏 ll 1 , 𝑏 ll 2 , ..., 𝑏 ll 𝑘 ], with 𝑏 ll 𝑖 = 𝑝 T 𝑖 𝑓 (𝑠 𝑡 ) + 𝑝 T 𝑖 (𝜂 − 1)𝑠 𝑡 + 𝜂𝑞 𝑖 + 𝑝 T 𝑖 𝑔(𝑠 𝑡 )𝑢 ll 𝑡(12)

In the above optimisation problem (12), 𝑢 cbf 𝑡 is optimised to evaluate the minimal correction required to ensure the safety of the system.

Decentralised CBF for CAVs

In the previous sections, the CBF ℎ(𝑠 𝑡 ) has been defined based only on the ego vehicle's state. The safety constraints defined in this section consider MAS dynamics because CAVs depend on the control decisions of other vehicles to ensure their own safety. These safety constraints are defined for pure CAV traffic. The extension of the safety constraints to mixed CAV traffic, consisting of vehicles with varying levels of autonomy and connectivity, is left for future work. In this section, specifications for decentralised CBFs are defined first (Section 4.1). Then, the multi-agent actor dependencies are defined for CAVs(Section 4.2). In the end, CBF formulations are defined based on the actor dependencies to comply with the specifications (Section 4.3).

Specifications

The following specifications are considered to formulate decentralised CBFs for CAVs:

1. Ensure the safety of all CAVs in a MAS. 2. Safe acceleration control to avoid collision with the preceding vehicle. 3. Safe steering control to avoid collisions during lane change manoeuvres. 4. Respect the CAV controller's physical limits.

Multi-agent actor dependencies

As CAVs can share their states with the surrounding vehicles, the state 𝑠 𝑡 of the ego vehicle constitutes its own ego states, 𝑠 e 𝑡 , and observed states, 𝑠 o 𝑡 , of observed vehicles. With this consideration, a dynamic CBF ℎ for two CAVs can be defined as

ℎ(𝑠 𝑡 ) = ℎ(𝑠 e 𝑡 ) + ℎ(𝑠 o 𝑡 )(13)

Notice that each term in the previously-defined optimisation constraint ( 12) is defined based on the state and the action variables from a single agent. For MAS, each term can be separated into variables associated with the ego vehicle * e and the observed vehicle * o as follows

𝐴 e 𝑢 e 𝑡 + 𝐴 o 𝑢 o 𝑡 ≤ 𝑏 e + 𝑏 o(14)

where 𝐴 e and 𝑏 e are equivalent to 𝐴 and 𝑏 ll (from( 12)), but evaluated using the state 𝑠 𝑡 and the action 𝑢 ll 𝑡 associated with the ego vehicle. Similarly, 𝐴 o and 𝑏 o are derived from the state and the action variables associated with the observed vehicle. The safe action for an ego vehicle can be obtained by optimising the minimum control correction 𝑢 e 𝑡 , assuming that the observed vehicle shares its state variables and safe control decisions. Therefore, the multi-agent constraint in ( 14) is modified to update the quadratic program (12)

The actor dependency exists between ego vehicle and observed vehicles as the term 𝑏 ma in the above equation (15) requires the observed vehicles to make their control decision, 𝑢 o 𝑡 , before the ego vehicle. Based on the ego vehicle's high-level behaviour, the observed vehicles are identified to be considered in CBF constraints.

If the ego vehicle is following a lane, its control action depends on the immediate leading vehicle within the communication range, as illustrated in Figure 3a. In this case, the longitudinal distance between the ego and the leading vehicle Δ𝑥 l must be constrained to ensure safety. If a vehicle in the adjacent lane intends to change lanes to the ego vehicle's current lane behind the immediate leading vehicle, the ego vehicle's decision depends on the adjacent vehicle, as shown in Figure 3b. In this case, the longitudinal and lateral distances, Δ𝑥 a and Δ𝑦 a , with the adjacent vehicle are constrained. As the ego While changing lanes, the ego vehicle depends on the actions of immediate leading vehicles in the current lane and the adjacent target lane. Similar to the previous case, longitudinal distance from the leading vehicle Δ𝑥 l is constrained. Moreover, the longitudinal and lateral distances from the adjacent vehicle, Δ𝑥 a and Δ𝑦 a , are constrained. This dependency is illustrated in Figure 3c.

Collectively, the safety of all the CAVs can be ensured by following the actor dependency as an individual CAV ensures safety with respect to its preceding vehicles. To honour this dependency, CAVs in a MAS are assumed to make control decisions sequentially in the decreasing order of their longitudinal position. Moreover, this actor dependency is applicable only to roads with two lanes. These limitations will be addressed in future work.

CBF formulation

CBF constraints for CAVs, CBF-CAV, are defined to ensure safe longitudinal and lateral motion without violating the physical control limits of the vehicles. This section defines CBFs for each type of safety constraint, along with the conditions for their applicability.

Longitudinal motion

The safety constraint for longitudinal motion allows the ego vehicle to maintain a safe headway with a preceding vehicle in the current lane. This constraint can be defined as, ℎ lon = Δ𝑥 l − 𝑥 safe (16) where Δ𝑥 l is the longitudinal distance from the rear end of the preceding vehicle and the front of the ego vehicle. The 𝑥 safe is the safe distance that must be maintained between two vehicles to ensure that the following vehicle has enough time to slowdown if the leading vehicle start slowing down abruptly. The safe distance threshold can be evaluated from the ego vehicle velocity 𝑣 e and time headway 𝜏 as shown below,

𝑥 safe = 𝜏 * 𝑣 e(17)

Both lane following and lane changing vehicles use this constraint, as the ego vehicle is expected to maintain a safe distance from the leading vehicle in all driving scenarios. Moreover, the leading vehicle must be within the ego vehicle's communication range to enforce this constraint.

Lateral motion

This constraint ensures safety when a CAV moves laterally to change lanes. During the lane change, a vehicle must maintain a safe distance 𝑥 safe with a leading vehicle in the same lane. This constraint can be enforced with a previously defined CBF (17). As the vehicle moves laterally, either a safe lateral distance 𝑦 safe or a safe longitudinal distance 𝑥 safe must be maintained with a leading vehicle in the adjacent lane. This constraint is defined as ℎ lat

ℎ lat = Δ𝑥 a 𝑥 safe + Δ𝑦 a 𝑦 safe − 1(18)

where 𝑥 safe is the same variable defined in equation ( 17), 𝑦 safe is a constant defined based on lane width 𝐿 𝑤 and vehicle width 𝑉 𝑤 to ensure comfortable lateral distance when the adjacent leading vehicle is moving in parallel.

𝑦 safe = 𝐿 𝑤 − 𝑉 𝑤(19)

A safe distance can be maintained with a vehicle in the adjacent lane with this constraint. Before changing a lane, this constraint ensures that the ego vehicle maintains a safe lateral distance from the adjacent vehicle. During the lane change, this constraint allows partial violations of lateral and longitudinal constraints while maintaining sufficient distance to avoid collision. During the execution of lane change manoeuvre, the partial violations allow a vehicle to gradually reduce the lateral distance Δ𝑦 a while gradually increasing the longitudinal distance Δ𝑥 a with the adjacent vehicle. The gradual increase in longitudinal distance ensures that a safe distance is maintained after completing the lane change manoeuvre. For example, if a vehicle in the adjacent lane is parallel to the ego vehicle, then the lateral safe distance must be maintained, Δ𝑦 a ≥ 𝑦 safe . In another case, if the adjacent vehicle is about to enter the ego vehicle's lane, then the ego vehicle must gradually increase the longitudinal distance, such that Δ𝑥 a ≥ 𝑥 safe when the adjacent vehicle enters the current lane. This constraint is applied to lane changing vehicles and lane following vehicles if they are obstructed by the adjacent leading vehicle changing lanes.

Control limits

Given that vehicle control inputs are subject to physical limitations, they must be constrained. The physical constraints are defined on the steering angle (𝛿), which is constrained within the range [-𝛿 max , 𝛿 max ]. This physical constraint is defined using two CBFs ℎ max 𝛿 and ℎ min 𝛿 . Note that the steering angle is one of the state variables, hence the constraints are defined using CBFs. The acceleration range of the vehicle is defined as [−𝑢 max 1 , 𝑢 max 1 ]. This physical constraint can be enforced by including it in the constraints of the optimisation problem defined in (12).

−𝑢 max 1 ≤ 𝑢 1 ≤ 𝑢 max 1 (21)

These constraints are applied to all CAVs, as they must be honoured in all scenarios.

Conclusion

The proposed CBF formulations, CBF-CAV, integrated with the behavioural layer with MARL lane change controller can have minimal impact on the efficiency, because the CBF constraints override the actions only when a vehicle is about to go towards unsafe states. Moreover, by restricting agents to the safe states, the MARL controller can be trained efficiently by exploring the safe states only. Furthermore, the actor dependencies are used to consider MAS dynamics in the CAV traffic. These constraints ensure both lateral and longitudinal safe motions for all CAVs in a traffic scenario.

The provided formulations are suitable for pure CAV traffic, where a lower safe distance can be used. In the future, this can be extended to mixed traffic with dynamic CBFs to maintain higher safe distances with human driven vehicles, assuming they take worst-case control decisions.