In tackling the complexities of vehicle trajectory planning at intricate trafic intersections, where prioritizing safety and eficiency is crucial, this research introduces an innovative model dubbed Adaptive Hierarchical Trafic Intersection Reinforcement Learning (AHTRL). This model is underpinned by a hierarchical deep deterministic policy gradient algorithm, which is structured into two principal layers: the upper layer is tasked with policy decision-making, while the lower layer focuses on the generation of precise waypoints. Steering and throttle adjustments are meticulously managed by a foundational PID controller, ensuring pinpoint accuracy in vehicular control. The model is further enhanced by integrating recurrent neural networks for the analysis of historical vehicle trajectory data, alongside a spatio-temporal variational autoencoder (ST-VAE) for the prediction of pedestrian future movements, thereby markedly improving interactive safety measures. Rigorous testing conducted on the Carla simulation platform has demonstrated the model's exceptional performance, outstripping existing methodologies across several critical metrics. Relative to the optimal baseline model, AHTRL has achieved a commendable 11.9% boost in total average rewards, a notable 13% increase in average transit velocity, and a significant 36% decrease in collision occurrences, afirming its dominance in ensuring safety and enhancing eficiency within the realm of complex intersection trajectory planning. These outcomes not only underscore the model's superior safety and eficiency but also its remarkable adaptability and practicality in navigating complex trafic scenarios. By melding advanced hierarchical reinforcement learning frameworks with cutting-edge deep learning technologies, this study substantially elevates the caliber of vehicle trajectory planning at convoluted trafic intersections. It paves the way for novel, eficacious solutions for fostering safe and eficient interactions within intelligent transportation systems. This contribution is not only academically innovative but also sets a robust foundation for real-world applications.
In modern society, with the acceleration of urbanization and the continuous increase in the number of vehicles, trafic intersections have become one of the most complex and challenging parts of the urban transportation system. Vehicle trajectory planning at trafic intersections is of great practical significance for improving trafic eficiency, reducing trafic congestion, and lowering the rate of accidents. With the development of autonomous driving technology, how to achieve safe and eficient vehicle trajectory planning in complex trafic intersection environments has become a key issue in the field of autonomous driving research.
In traditional vehicle trajectory planning research, the vast majority of methods depend on accurate
environmental models and predefined rules [
In recent years, learning-based methods, especially reinforcement learning, have become efective
means for solving the problem of vehicle motion trajectory planning at trafic intersections.
Reinforcement learning, by interacting with the environment, learns the mapping from observed states to actions
taken, demonstrating strong potential [
Addressing the challenges mentioned above, this chapter introduces a hierarchical reinforcement learning framework, AHTRL, specifically designed for the trafic intersection scenario. The core idea of the AHTRL method is to decompose the decision-making problem in complex trafic intersections into multiple sub-problems and solve them hierarchically, thereby enhancing the overall eficiency and stability of decision-making. Moreover, this study pays special attention to the safe interaction between vehicles and pedestrians. To accurately predict pedestrians’ future trajectories and plan the vehicle’s driving path based on this, the chapter utilizes a Spatio-Temporal Variational Encoder (ST-VAE) to process the historical trajectory data of pedestrians.
In terms of technical implementation, this study employs a hierarchical approach known as Hierarchical Deep Deterministic Policy Gradient (HDDPG). The model is structured into three layers (as shown in Figure 1), with the top layer, the selection layer, responsible for making vehicular action decisions, such as stopping, turning left, turning right, and decelerating. These decisions are based on an analysis and evaluation of the current state of the vehicle, environmental conditions, and objectives. For instance, when approaching an intersection, the selection layer decides whether to continue straight, decelerate, or stop, considering trafic rules, the condition of the intersection, and vehicle objectives. This layer’s decisions provide guidance for the vehicle, ensuring that its actions comply with safety and eficiency requirements. Guided by high-level decisions, the trajectory planning layer then generates specific waypoint trajectories that detail how the vehicle should move from its current position to the destination, considering the vehicle’s dynamic limitations, road conditions, and obstacle avoidance needs to ensure the trajectory is both safe and practical. In this way, the trajectory planning layer bridges high-level policy decisions and low-level execution controls, providing a clear and feasible path. Finally, the Proportional-Integral-Derivative (PID) controller operates at the trajectory tracking layer, ensuring the vehicle accurately follows the planned waypoint trajectory by dynamically adjusting the vehicle’s state (such as speed and direction) to minimize deviations. The PID controller adjusts its three parameters (proportional, integral, derivative) in response to any deviations from the trajectory, ensuring the vehicle closely follows the predetermined path. Moreover, by analyzing pedestrians’ historical trajectories and predicting their future movements, this study can plan vehicle trajectories more accurately, ensuring safe and efective interaction between vehicles and pedestrians at trafic intersections.
In summary, this paper efectively addresses the vehicle trajectory planning problem in the complex scenario of trafic intersections by proposing a hierarchical reinforcement learning framework specifically designed for this context. Simulation results demonstrate that, compared to existing methods, the framework proposed in this chapter performs better in adapting to the complexity and uncertainty of trafic intersections, successfully reducing merging time and collision rates under the premise of ensuring safety, and generating smoother trajectories. These achievements not only showcase the application value of hierarchical reinforcement learning in the field of autonomous driving but also ofer new perspectives and methodologies for future trajectory planning research in complex trafic environments. The main contributions are as follows: • For the complex scenario of trafic intersections, this paper introduces a hierarchical decision
model, AHTRL, which efectively solves the vehicle motion trajectory planning problem. This approach decomposes the complex decision-making problem into multiple sub-problems and solves them layer by layer, significantly improving the eficiency and stability of the decisionmaking process. • This study incorporates a Spatio-Temporal Variational Autoencoder (ST-VAE) for processing pedestrian historical trajectory data, with the objective of precisely forecasting future pedestrian movements. Leveraging these predictions, the AHTRL model meticulously crafts vehicle driving paths to ensure a safe interaction between vehicles and pedestrians. This approach not only enhances the precision of trajectory predictions but also furnishes vehicles with robust decisionmaking support for autonomous navigation in intricate environments. • This research employs a hierarchical approach using Deep Deterministic Policy Gradient (DDPG) methodology, coupled with a Proportional-Integral-Derivative (PID) controller, to realize a comprehensive process from high-level policy decision-making through to detailed trajectory planning and precise trajectory tracking. This technical implementation not only elevates the accuracy of trajectory planning but also ensures the smoothness and safety of the path. • Testing on the Carla simulation platform has demonstrated that the AHTRL model surpasses existing methodologies in both eficiency and safety. Compared to the optimal baseline model, it achieved an 11.9% increase in total average rewards, a 13% improvement in average transit speed, and a 36% reduction in collision rates. These results validate the superior performance and practical value of the AHTRL model in complex intersection trajectory planning.
This section provides a systematic overview of the existing research in the vehicle trajectory planning domain, with a particular focus on traditional methods, the application of reinforcement learning, and the progress in hierarchical reinforcement learning frameworks. These discussions lay the theoretical and technical groundwork for this paper, aiming to ofer robust background support for further exploration and resolution of vehicle trajectory planning challenges.
In the field of autonomous driving, trajectory planning is one of the key technologies ensuring safe
and eficient vehicle operation. In recent years, non-reinforcement learning (NRL) methods have
made significant progress in addressing trajectory planning problems. These methods can be broadly
categorized into four groups: classical planning methods, heuristic-based methods, supervised learning
approaches, and statistical methods. Classical planning methods, such as Rapidly-exploring Random
Trees (RRTs) [
In summary, non-reinforcement learning methods for trajectory planning each have their advantages and limitations. Classical planning methods and heuristic-based approaches are efective in specific scenarios but may lack flexibility and generalization capability. Supervised learning and statistical methods can handle more complex scenarios but rely heavily on large amounts of data and computational resources. In contrast, reinforcement learning methods ofer a more dynamic and adaptive trajectory planning solution through learning from interactions with the environment. Nonetheless, in-depth research into non-reinforcement learning methods remains crucial for understanding and improving the trajectory planning of autonomous vehicles.
Although non-reinforcement learning methods can directly learn from human driving behaviors, their
reliance on large amounts of manually annotated data and the uncertainty brought about by diferences
in decision-making among drivers limit their application in complex tasks. To reduce the dependency on
labeled data, researchers have turned to reinforcement learning (RL) for autonomous decision-making
and planning. The advent of Deep Reinforcement Learning (DRL), which combines deep learning
techniques, has shown immense potential in handling complex decision-making and planning problems,
especially achieving breakthrough progress in areas like intelligent gaming, natural language processing,
and autonomous driving. DRL is capable of exploring a wide range of possibilities, including hazardous
scenarios, and has the potential to achieve performance beyond human capabilities. The study by Mnih
et al. [
Reinforcement learning ofers an efective approach for vehicle trajectory planning, but to fully leverage this technology’s potential, challenges such as low sample eficiency and handling complex tasks must be overcome. Future research directions include exploring more eficient learning algorithms and integrating advanced perception and decision-making mechanisms to achieve efective trajectory planning in even more complex environments.
Hierarchical Reinforcement Learning (HRL) simplifies the learning process by dividing the overall problem into a hierarchical structure of multiple subtasks. Each subtask is assigned specific objectives and strategies, and these subtasks are organized hierarchically, with higher-level subtasks providing guidance and context to lower-level ones. This layered approach allows agents to focus on narrower problem scopes, reducing the complexity of the learning task and making the problem more solvable.
Chen et al. [
Hierarchical Reinforcement Learning (HRL) has garnered attention for its significant advantages in simplifying complex tasks and improving learning eficiency. However, it faces considerable challenges in ensuring pedestrian safety, a critical area. Particularly in unpredictable trafic intersection scenarios, HRL shows certain limitations in dealing with dynamic environments involving pedestrian interactions. The model’s insuficient predictive ability often struggles to accurately capture pedestrian movement trajectories and intentions, limiting autonomous vehicles’ capacity for safe and eficient passage in complex trafic situations. Moreover, the generalization ability of HRL strategies is also challenged. Faced with new trafic scenes or previously unseen pedestrian behaviors, even well-trained models may falter, struggling to respond appropriately, which compounds the dificulty of ensuring pedestrian safety. This limitation in generalization, stemming from HRL’s inherent hierarchical structure, restricts its adaptability in unknown environments. Thus, despite HRL’s notable advantages in handling complex tasks, ensuring pedestrian safety in critical scenarios like trafic intersections remains fraught with multiple challenges. Future developments should focus on enhancing the model’s predictive accuracy regarding environmental changes and strengthening strategy generalization capabilities, aiming to ensure pedestrian safety without compromising eficiency and fluidity in complex trafic environments. This requires not only deepening technical research but also continuously exploring new ideas and methods to overcome existing limitations, pushing autonomous driving technology towards higher levels of safety and intelligence.
Figure 2 depicts a Hierarchical Reinforcement Learning framework tailored for vehicle motion planning at trafic intersections, named AHTRL. The AHTRL model is divided into three levels, each performing diferent trajectory decision-making and planning tasks. Both the high-level selector and the low-level planner utilize the Actor-Critic structure of the Deep Deterministic Policy Gradient (DDPG). The high-level selector undertakes key decision-making tasks, such as stopping, turning, and decelerating, based on a comprehensive analysis of the current situation of vehicles and pedestrians as well as the surrounding environment. The lower-level trajectory planning layer generates precise waypoint trajectories according to decisions made by the top level, taking into account the vehicle’s dynamic characteristics, current trafic conditions, and obstacle avoidance needs, ensuring the safety and practicality of the trajectory. The trajectory tracking layer ensures that the vehicle can accurately follow the planned waypoints through a PID controller. Additionally, the framework integrates a Spatio-Temporal Variational Autoencoder (ST-VAE) model, specifically for predicting pedestrian future trajectories, which is crucial for ensuring safe interactions between vehicles and pedestrians at trafic intersections.
The ST-VAE (Spatio-Temporal Variational Autoencoder) model is dedicated to predicting the future movement trajectories of entities within a scene. In a complex setting containing N pedestrians, the ST-VAE model can predict their future location distribution over a future time span H by analyzing each pedestrian’s spatial position over a time span T. The ST-VAE model particularly focuses on extracting features from pedestrians’ social behaviors and independently predicting each pedestrian’s future trajectory. This model is adaptable to scenarios of varying scales, especially in urban scenes where dynamic changes occur rapidly and the pedestrian’s surrounding environment frequently changes. Even when a pedestrian’s local neighborhood is dificult to continuously monitor, the ST-VAE model can accurately predict their future positions. This predictive capability not only aids in enhancing the eficiency of autonomous driving and intelligent surveillance systems but also provides new perspectives and tools for understanding and analyzing crowd dynamic behaviors.
Figure 3 showcases the overall architecture of the model proposed in this study, centrally featuring a
time-based variational autoencoder integrated with the Complex Gated Recurrent Unit (CGRU) structure
designed in previous research [
( +1: + |1: ) = ∏︁ ( + | +1: + − 1, 1: ).
=1
To efectively capture the complex interactions between pedestrians and other trafic participants in crowded scenes, the ST-VAE model employs a Graph Attention Network (GAT). Trafic participants within the scene (such as pedestrians, trafic lights, vehicles, and crosswalks) are represented as nodes in a graph, with edges between nodes indicating interactions between pedestrians and their environment. GAT dynamically assigns importance weights to diferent nodes, thereby efectively aggregating information from neighboring nodes. This enables the model to understand the nuanced patterns of pedestrian interactions within social environments more intricately. This approach not only considers the influence of other pedestrians but also takes into account the impact of the surrounding environment on pedestrian behavior, addressing the limitations of traditional LSTM models in capturing such complex social interactions. Recent studies have shown that in scenarios where the behaviors of each trafic participant are interconnected with surrounding pedestrians and environmental factors, GAT models provide a more comprehensive perspective for pedestrian trajectory prediction by mapping these interactions onto a graph structure.
By employing a Graph Attention Network (GAT), we can acquire information on all pedestrians from time step 1 to T. To synthesize the information of each pedestrian at every time point 1: , the ST-VAE model utilizes a Long Short-Term Memory network (LSTM) to aggregate information across time steps (as indicated in Equation 2).
1: = LSTM(ˆ1 , ˆ2 , . . . , ˆ )
Where ˆ represents the comprehensive hidden state of the ℎ pedestrian at time , which is obtained after processing (the state information of the ℎ pedestrian at time step ) through two layers of graph attention.
Another key feature of the ST-VAE model is its ability to generate stochastic predictions, enhancing the model’s flexibility and the randomness of its forecasts. This is achieved by introducing latent (1) (2)
Equation 3). variables during the sequence generation process, each of which updates its state via the CGRU network to reflect the complexity of pedestrian movement states and environmental interactions (as indicated in ︁( ˆ = ⃗ ︀( , )︀ , ˆ)︁
The training process of the ST-VAE model follows the standard VAE training objective of maximizing the Evidence Lower Bound (ELBO), while also considering the potential accumulation of errors in the final trajectory generated through displacement sequences. By employing the reparameterization trick with Gaussian distributions and adjustments to the training loss, the model can compensate for predictive errors from previous time steps, thereby enhancing the overall accuracy of the predictions (as indicated in Equation 4). [︃ 1 +
∑︁ E = +1
E∼ (·| ,^)E∼ (·| ,^) ⎣⃦ ⃦ −
− ⎡ ⃦ ⃦ ⃦
⃦ ∑︁ = +1 ⃦ ⃦ 2⎤⎤ ⃦⃦ ⎦⎦ +( | , ˆ) − (| ˆ).
Where , , and are parameterized through network parameters, forming Gaussian distributions.
In designing a hierarchical reinforcement learning model for trajectory planning, this paper utilizes two independent network layers: a high-level selection network and a low-level planning network. This layered approach allows for specific optimizations at diferent decision-making levels and facilitates modular learning of complex tasks.
A. The high-level selection network
The high-level selection network is responsible for formulating long-term strategies and objectives, functioning to generate sub-goals that guide the lower-level planning network. This network, by observing the current environmental state , generates a high-level action or sub-goal to indicate the macro objective the vehicle aims to achieve. In the context of autonomous driving, these sub-goals could be specific intersection behaviors such as "stop," "turn left," "turn right," or "decelerate." The selection network’s responsibility for generating sub-goals can be represented by the following equation: = (| ) Where represents the parameters of the selection network’s policy, and is the current state.
The objective of the policy is to maximize the expected return, and the value function of the selection network is represented as:
The policy parameters are updated through gradient ascent: And updated using the actor-critic method: (, ) = E [︃ ∑︁ = − |, =
]︃ ← + ∇ (, ) ← + ∇ (, , (, )) (3) (4) (5) (6) (7) (8) (9) B. A low-level planning network
The low-level planning network receives the sub-goal passed down from the selection network and generates specific actions based on this. This layer’s network focuses on calculating short-term, specific waypoint trajectories that adhere to the vehicle’s dynamic constraints, take into account road conditions and obstacle avoidance requirements, to ensure the safety and practicality of the trajectory. The planning network is responsible for outputting actions under the guidance of the given sub-goal , which can be represented by the following equation: = (, | ) (10) Where represents the parameters of the planning network’s policy.
The objective of the policy is to maximize the expected return, and the value function of the planning network is represented as follows:
(, , ) = E [ + (+1, , +1)|, , = ] The policy parameters are updated through gradient ascent:
← + ∇ (, , ) Here, is the discount factor, is the learning rate, and represents the immediate reward. And updated using the actor-critic method:
The network structures of the high-level selector and the low-level planner, as shown in Figure 4, take (i.e., 2D LIDAR images and BEV semantic images processed through a 32×256 preprocessing layer) as environmental inputs. The network architecture includes a self-attention mechanism, an input layer (MLP), a Long Short-Term Memory network layer (LSTM), an output layer (MLP), and a dense layer used for generating action behaviors. The self-attention layer aims to identify the most critical parts within the fused features for the current task, thereby enhancing the precision of the high-level selector and the low-level planner in the goal generation process. The input layer (MLP) further processes the feature vectors, typically integrating multiple fully connected layers and nonlinear activation functions to refine key features and construct a basis for decision-making. The LSTM layer, essential for handling time-series data, is crucial for capturing time-dependent states and actions, such as the vehicle’s historical movement trajectories. After processing through the LSTM layer, data flows to the output layer (MLP), which maps features to the action space, laying the groundwork for generating deterministic actions. Finally, the dense layer transforms output layer data into continuous action values, directly guiding the vehicle’s movements in the environment.
The design of the reward mechanism is based on several factors, including longitudinal speed, penalties
for collisions (in this chapter, collisions between vehicles and pedestrians are specifically identified
with collision and given the maximum penalty), lane deviation, large steering angles, speeding, and
significant lateral acceleration. It is designed based on a similar environmental setup used by Chen et al.
[
In the work of this chapter, the corresponding weights for each factor are designed as follows: 1 = 1000, 2 = 200, 3 = 1, 4 = 10, 5 = 1, 6 = 5, 7 = 0.2, 8 = − 0.1.
This paper utilized the CARLA simulator [
In the initial phase of the experiments, this paper conducted a random policy for 10,000 steps on each map to initialize the experience replay bufer. Subsequently, 30,000 steps of training were carried out, enabling the agent vehicle to efectively plan driving paths, avoid collisions, maintain lanes, and appropriately interact with other vehicles and pedestrians in various intersection environments. Moreover, to enhance the vehicle’s performance in pedestrian-dense trafic intersections, this study added 100 pedestrian models at each intersection. On top of the original 30,000 steps of training, an additional 10,000 steps of specialized training focused on vehicle-pedestrian interactions were conducted.
This paper particularly emphasizes the interactive planning between vehicles and pedestrians by introducing pedestrian models with diverse behavior patterns. It designs algorithms that enable RLcontrolled vehicles to recognize pedestrians, predict their actions, and take appropriate measures to avoid them when necessary, ensuring pedestrian safety. Through this series of experimental setups and training, the approach not only improves the vehicle’s motion planning capabilities at intersections but also significantly enhances its safety performance in pedestrian-dense environments.
Ultimately, this paper compared the proposed method with several baseline algorithms, validating its efectiveness in complex trafic environments, particularly in ensuring the safe coexistence of vehicles and pedestrians. This research not only showcases the application potential of hierarchical reinforcement learning technology in the field of autonomous driving but also provides valuable insights and practical guidance for the safe operation of autonomous driving systems in complex urban environments.
In the experiments conducted in this paper, the proposed AHTRL model (hereafter referred to as
Our_HRL) was compared with the original Deep Q-Network (hereafter referred to as DQN), H_DQN
[
1. DQN (Deep Q-Network): A cornerstone algorithm in deep reinforcement learning that combines Q-learning with deep neural networks to handle high-dimensional state spaces. Comparing with DQN allows for evaluating the efectiveness of hierarchical approaches against a foundational deep learning-based method.
2. H_DQN [
3. DDPG (Deep Deterministic Policy Gradient): A model-free, of-policy actor-critic algorithm that can operate over continuous action spaces, making it highly relevant for real-world applications like autonomous driving. Comparing with DDPG showcases the benefits of hierarchical modeling in environments where precise control over actions is crucial.
4. atHRL [
By comparing Our_HRL with these methods, the paper aims to underscore the improvements in learning eficiency, decision-making quality, and adaptability to complex scenarios brought about by the proposed hierarchical reinforcement learning model, particularly in the context of autonomous vehicle navigation and pedestrian safety.
Figure 5 displays the rewards obtained during the training process by diferent algorithms, where
the reward value comprehensively considers penalties for various factors including collisions with
pedestrians, other types of collisions, lane deviations, speeding, as well as large steering angles and
high lateral accelerations. Thus, the comparison of reward values efectively reflects the performance of
each method. The results indicate that after 30,000 steps of training, the strategy proposed in this study
achieved the highest reward among all four comparative methods. Figure 6 further demonstrates that
the method proposed by this study performs exceptionally well in pedestrian-dense trafic intersection
scenarios, obtaining the highest reward after 10,000 steps of training. Table 1 shows that the method
proposed by this study surpasses all other comparison methods in terms of average reward and average
speed. While the average reward reflects the overall performance of the agent, comparing average
speeds helps to verify the rationality of the vehicle driving strategy. Compared to the optimal baseline
model atHRL [
The hierarchical reinforcement learning method proposed in this paper was compared with the original DQN and DDPG algorithms. The results show that the use of a hierarchical planner significantly improved the performance of the reinforcement learning agent in specific urban driving scenarios. Compared to the original DDPG, the average reward increased by 73%, and the average speed of vehicle passage increased by 70.3%. This performance enhancement is not only reflected in a higher overall average reward, indicating that the agent can efectively avoid collisions and ensure safe driving in the simulated environment, but also in a faster average driving speed, demonstrating the agent’s enhanced robustness in handling complex scenarios. The introduction of a high-level decision planner enhanced the stability of motion control and reduced the risk of collisions. In contrast, learning low-level control commands directly from observations could lead to unstable control and dificulties in learning strategies for diferent tasks. Additionally, the method proposed in this paper performed better than H_DQN, which has a similar trajectory planning structure. Although H_DQN is efective for simple decisions, it is insuficient for more complex urban driving task combinations. Compared to the traditional hierarchical DDPG model, atHRL, the average reward increased by 11.9%, and the average speed of vehicle passage increased by 13%. This indicates that by incorporating pedestrian trajectory prediction, the method in this chapter more efectively managed interactions between pedestrians and vehicles at trafic intersections. Furthermore, the introduction of a self-attention network into the actor-critic model enabled the vehicle to focus on key environmental elements, thereby improving the overall efectiveness of policy learning, leading to superior results in complex urban driving scenarios.
Safety analysis of vehicle trajectory planning is crucial in the development of autonomous driving systems. By comprehensively assessing the system’s safety performance under various conditions, this paper gains a deep understanding of the system’s behavior and promptly identifies potential safety hazards. This provides essential guidance for system design, such as determining the appropriate planning algorithms, route selection strategies, and parameter settings. Moreover, safety analysis supports the optimization of planning strategies, aiding in the identification and resolution of existing issues and shortcomings within the system. Furthermore, the results of the safety analysis ofer targeted recommendations and support for decision-making, thereby ensuring the safe operation of autonomous driving systems in all circumstances. In summary, safety analysis of vehicle trajectory planning is an indispensable part of the autonomous driving system development process, vital for ensuring the system’s safety performance, guiding system design and optimization, and supporting decision-making. To evaluate vehicle safety, this paper employs two key metrics: collision rate and success rate. 1. Collision Rate: The percentage of test events in which collisions occur.
2. Success Rate: The percentage of test events where the test vehicle successfully completes its trajectory from start to end without any collisions.
The evaluation of the trained policy consists of 500 test episodes that cover the vehicle safely navigating through various types of trafic intersections, including roundabouts, crossroads, T-junctions, and intersections without trafic lights.
As shown in Table 2, across 500 test scenarios, the algorithm proposed in this paper demonstrated a lower collision rate and a higher success rate. Compared to the best baseline model atHRL, the collision rate decreased by 36.3%. The Our_HRL(AHTRL) algorithm exhibits significant advantages in vehicle trajectory planning over traditional reinforcement learning algorithms. First, Our_HRL(AHTRL) can handle continuous action spaces, producing smoother and more natural trajectories compared to DQN, which enhances driving comfort and safety. Second, by incorporating a hierarchical structure and recurrent neural networks, Our_HRL (AHTRL) achieves more flexible and precise decision-making and more accurately predicts pedestrian behavior compared to DDPG, improving safety when navigating through trafic intersections. Compared to hierarchical DQN (H_DQN) and hierarchical DDPG (atHRL), Our_HRL (AHTRL) not only calculates continuous target waypoints but also adopts a mixed reward mechanism and reward-driven exploration strategy, thereby improving learning eficiency and convergence speed. In summary, the low collision rate and high success rate demonstrated by Our_HRL (AHTRL) in navigating trafic intersections are attributed to its adaptability in continuous action spaces, the stability of deterministic policy gradients, the eficiency of its hierarchical structure, the predictive capability of recurrent neural networks, and the learning eficiency of its mixed reward mechanism.
This paper addresses the vehicle trajectory planning problem at complex trafic intersections by proposing a hierarchical reinforcement learning-based model, AHTRL. By decomposing the decision-making process, predicting pedestrian behavior, and integrating the pedestrian trajectory prediction model, ST-VAE, the model efectively enhances vehicle trajectory planning performance in variable trafic environments, significantly reducing collision rates and improving safety and eficiency. Experimental results demonstrate the method’s exceptional performance across various trafic intersection environments, particularly in pedestrian-dense trafic intersection scenarios, where it significantly enhances safety and reduces collision risks. These research achievements not only highlight the application value of hierarchical reinforcement learning in autonomous driving technology but also provide new perspectives and methods for future research on trajectory planning in complex trafic environments.
In future research, plans are in place to optimize and extend the AHTRL model proposed in this paper from multiple dimensions. Firstly, eforts will be dedicated to the optimization and improvement of the algorithm by exploring more eficient training strategies and novel reward mechanisms to enhance the algorithm’s convergence speed and stability while fine-tuning the balance between safety and eficiency. Secondly, considering the complexity of real-world trafic environments, research on multiagent collaboration will become a focus. This includes studying interaction strategies between vehicles as well as between vehicles and pedestrians, thereby enhancing the coordination and eficiency of the overall trafic system. Furthermore, transferring the model from simulation environments to real-world applications and conducting tests with actual vehicles will be a critical step in verifying the model’s practicality and robustness. At the same time, exploring the model’s cross-scenario generalization ability aims to develop more versatile trajectory planning models adaptable to diverse trafic environments and geographical locations. Finally, considering the social impact of autonomous driving technology, future work will also address ethical and responsibility issues in artificial intelligence, ensuring that technological development adheres to moral standards while safeguarding the rights of users and the public. These research directions will not only drive technological progress in the field of autonomous driving but also contribute to realizing a safer and more eficient autonomous driving future.
This work is supported by the National Natural Science Foundation of China (62166043), the Joint Funds of the National Natural Science Foundation of China (U1603262), and the "Intelligent Information R&D Project" (Project Number: 202104140010). We thank all anonymous commenters for their constructive comments.