Autonomous robots and vehicles are expected to become an integral part of our environment soon. Unsatisfactory issues (esp. for path planning) regarding interaction with existing road users, performance in mixed-trafic areas, and lack of interpretable behavior remain key obstacles. To address these, we present a physics-based neural network, based on a hybrid approach combining a social force model extended by group force (SFMG) with Multi-Layer Perceptron (MLP) to predict pedestrian trajectories considering its interaction with static obstacles, other pedestrians, and pedestrian groups. We quantitatively and qualitatively evaluate the model concerning realistic prediction, prediction performance, and prediction "interpretability". Initial results suggest that, even when solely trained on a synthetic dataset, the model can predict realistic and interpretable trajectories with better than state-of-the-art accuracy.
There has been growing research interest in autonomous technologies like autonomous vehicles, service robots, goods carriers, and surveillance robots. It is generally expected that such autonomous entities will soon be a part of our daily environment. However, before this, several hard issues remain to be resolved. Firstly, their performance in mixed-trafic zones and interaction with existing road users (e.g. cars, pedestrians, cyclists) requires further inspection [1], [2]. Secondly, uncertainty or lack of interpretability in their motion behavior and decisions make them less "socially acceptable" [2]. Again, in a mixed-trafic zone, autonomous robots/vehicles need to predict the future trajectories of other road users (cars, pedestrians, cyclists, etc.) to plan their trajectories and navigate safely. However, doing so is a complex task due to overlapping interactions among road users, road user groups, and static obstacles.
Figure 1 shows an example scenario. Service robot R1 is placed in a mixed-trafic zone, trying to navigate through other road users to reach its destination X. On its path, it will encounter cars (C2), pedestrians (P1, P2, P3, and P4), and static obstacles (e.g. trees, lamps). For that, it may need to predict the trajectories of these other road users and plan its path based on that. However, the behavior of an individual road user can also depend on other factors present in the environment. Factors like interaction with static obstacles, other road users, and (if the road user is traveling in a group) other group members. So, the robot/vehicle has to consider all these factors while predicting other road users’ trajectories and planning its path.
As pedestrians are vulnerable road users and form the majority in most mixed-trafic zones, modeling pedestrian behavior warrants high importance. This task entails estimating a pedestrian’s future motion within seconds based on its previous motion and environment-related information.
The main contribution of this paper is a novel pedestrian motion prediction model, a prerequisite for autonomous robot/vehicle path planning, which considers pedestrian interaction with static obstacles, other pedestrians, and pedestrian groups. We take a hybrid approach combining the classical social force model (SFM) [3] extended by pedestrian group behavior modeling [4], with a neural network for predicting pedestrian motion.
The remainder of this paper is arranged as follows: Section 2 reviews related work on trajectory prediction. Section 3 then briefly discusses SFMG; our model architecture is described in Section 4, followed by its evaluation in Section 5. Section 6 concludes and discusses future research opportunities.
Broadly, road user trajectory prediction algorithms can be divided into theory-based or physicsbased approaches and learning-based approaches. The first group of approaches uses domain/theoretical knowledge and/or physics-based models to model road user motion. The social force model [3] and its various extensions ([6], [7], [8], [4], [9]) are widely used for this purpose. At its core, SFM defines pedestrian motion as being motivated by external forces exerted upon it by static obstacles and other road users. These methods can model road user motion realistically, and they are interpretable. However, the complexity of estimating model parameters hinders their scalability, and they require too much explicit expert involvement, which in turn restricts their suitability to include new road user types [10].
Learning-based approaches using machine learning algorithms, especially artificial neural networks (ANN) have the potential to overcome the above-mentioned issues. With a suficient amount of relevant data, a machine learning algorithm can be trained to estimate the underlying motion dynamics of a road user. Diferent types of neural network architectures have been used to predict pedestrian trajectory, including Multi-layer Perceptron [11], Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) ([12], [13], [14]), Conditional Variational Auto-encoders [15], Generative Adversarial Networks [16], Graph Neural Networks [17], and Graph Attention Networks [18] and so on. However, the size of the neural network needed to estimate complex human behavior and the number of training samples needed jointly become a major bottleneck. Moreover, neural networks tend to be like "black boxes", lacking explainability. Explainability is required for autonomous entities to analyze their behavior, attribute responsibility, and engender trust [19]. The lack of a prior model in an ANN makes explainability hard or nearly impossible. There have been some attempts to combine theory-based and learning-based approaches to avail the advantages of both and consequently design a "Grey-box" model [20].
Johora et al. [21], proposes combining GSFM, an agent-based model, and LSTM-DBSCAN, a learning-based model, in a hierarchical manner, i.e. GSFM takes over when LSTM-DBSCAN predicts conflicting trajectories. Kim et al. [ 22], combines an ANN with a cellular automata model to estimate trafic congestion. Here, an ANN helps to estimate the cellular automata model’s parameters. Karpatne et al. [23] suggests a new paradigm of approaches that aim to combine theory-based knowledge with data-driven approaches and avail subsequent advantages. Antonucci et al. [24], suggests a combination of SFM and MLP to predict pedestrian trajectories. Here, the neural network architecture has been designed in such a manner that it estimates SFM parameters. This work [24] can estimate the acceleration force towards the desired destination and repulsive forces exerted by the nearby obstacles using two diferent MLPs. Interestingly, the second MLP consists of both the acceleration force part and repulsive force part together in a single network, jeopardizing its interpretable nature. Moreover, the authors do not consider interaction with other road users. Another approach to combining SFM and neural networks to design a diferentiable simulation model is introduced in Kreiss et al. [ 25]. Here, the neural network is used to estimate the interaction potentials. Pedestrian interaction with obstacle/s and other pedestrian/s is considered by the authors. The authors did not test the model [25] on real data. Noticeably, none of these works consider interactions among pedestrians and pedestrian group/s, although pedestrian groups contribute toward 70% of the road users [26].
Therefore, in this work we combine SFMG and neural network to model pedestrian motion and, consider interactions with obstacles, other pedestrians, and pedestrian groups. Here, SFMG helps understand and model/predict a pedestrian’s motivation behind making a certain motion behavior (i.e. through corresponding forces). The neural network architecture mimics the SFMG equations, and it is solely used to estimate the equation parameters. SFMG serves as a prior model for the neural network. Thus, the overall approach retains the social force model’s interpretability. Again, we employ individual networks to estimate individual forces and combine them in a modular manner to get total force. Refer to Section 4 for further details on the model architecture.
In this section, we briefly discuss the social force model [ 3] to understand how we can incorporate it with Multi-Layer Perceptron (MLP). Suppose, a certain pedestrian is moving towards its destination→− at a certain desired velocit→y− () := → − () at time t. Its motion path can be disturbed by static obstacles in the environment (e.g. walls, trees, etc.) and other road users. Then, according to SFM [3], the total force acting upon is defined as:
→− () =→− + ∑︁→− + ∑︁→− + ∑︁→− + Again, the acceleration force towards the destination→,− is defined as:
→− = (→ − () − →− ())/ where, 1 = + (1 − )/2 and 2 = (1 − )/2 are constants. (1) (2) (3) (4) (5) (6)
Here→,− () is the current velocity, is relaxation time an→d− () = →→||−− −−→→−− || is the desired direction→.− an→d− denote next position and current position of pedestrian respectively. A pedestrian’s tendency to keep a safe distance from static obstacles is denoted by the repulsive forc→e− as:
→− = − ()/→−
Here, B denotes a static obstacle, and is the interaction strength between and obstacle B. an→d− are the distance and the direction unit vector between and B respectively. The repulsive force between and another pedestrian is denoted by:
→− = − ()/→− where, an→d− are distance and unit direction vector between and . represents the anisotropic behavior of a pedestrian, and it is defined as:
= + (1 − )(1 + cos ( ))/2 Here, is a constant and
is the angle between the motion direction of pedestrian and the vector pointing in the direction of to . i.e. cos ( ) =→− (→).− (). Combining equation 4 and 5, and simplifying it, we get: →− = (1− ()/→−
+ →2− →− →− ()− ()/ )
The attractive force towards points of interest (e.g. friends, street artists, etc.) is given by →− . We ignore this force in our work to avoid further complexity. SFM has been extended by [4], [27] to include pedestrian group interaction by introducing the group forc→e− as: →− =→− +→−
In the above equation→,− denotes the pedestrian’s desire to keep other group members within his/her field of view (FOV).→− is an attraction force representing the pedestrian’s motivation to maintain group coherence. The term→s− an→d− can be described as: →− =
→− = * →*− equation 1 becomes:
Here, is the minimum rotation angle. and are global strength parameters→.− is the desired velocity of any pedestrian in group →.− ( , ) represents the normalized unit length vector between and group centroid . Adding→− and omitting→− 0, (7) (8) (9) (10) →− () =→− + ∑︁→− + ∑︁→− +→− +
We use the force equations of the social force model with pedestrian group behavior modeling described in this section to design our pedestrian motion prediction model architecture.
We aim to design our neural network based framework in such a way that it estimates the total force acting upon a pedestrian according to Equation 10. Equation 10 aggregates four individual forces, namely acceleration force→(− ), repulsion from boundaries→(− ), repulsive force from other pedestrians→(− ) and group force→(− ). We employ four individual networks to estimate these four forces separately. We call these individual networks as modules. A final network combines these individual modules to give final force ( →− ()/). Figure 2 shows the overall model architecture and highlights the individual modules. The model specific hyperparameters (e.g. activation function, learning rate, etc.) were chosen empirically. Further details are presented in Section 4.2 and Section 4.3.
We formulate our trajectory prediction problem similar to standard approaches used in the literature [17], [28]. Given the trajectories of all pedestrians in a scene, we need to predict their future trajectories. Let X = [1, 2, ..., ] be the current trajectories of all pedestrians present in the scene. We need to predict future trajectories, ˆ = [ˆ1, ˆ2, ..., ˆ ]. Here, for a certain pedestrian , the given trajectories are = [ , ] at time-steps = 0, 1, 2, ..., . The real or ground truth future trajectories for are = [ , ] at time-steps = + 1, ..., . Δp(t) D(t) ՜ ՜ ՜ ՜ net3_9 net3_8 ሱۛۛሮ ( ) * ՜ (a) (c) net3_1 ՜ net1 ՜ net3 ՜ ՜ (b) net2 networks corresponding to the forces in Equation 10. Note, net3 contains nine instances like net3_1.
net1: Acceleration
This module estimates the goal-directed acceleration force→(− ) as defined by Equation 2. The model architecture is designed according to Equation 2. This module is largely based on [24]. The module inputs are n previous positions (p) of the pedestrian , till the current time-step t. The position values are normalized by subtracting the first position value of the window ((− )) to avoid spatial bias. The new position values are termed as ∆ ().
net1 consists of two parts: one sub-network to estimate the instantaneous velocit→y− () and second sub-network estimates the desired velocity →0− (). The inputs to the networks are ∆ () and () respectively. Here () refers to the instantaneous velocity magnitudes, and it is calculated as: () = [||∆ ′ ()1||, ...., ||∆ ′ () − 1||]. Where, ∆ ′ () = ∆ () − ∆ ( − 1).
The first sub-network consists of two MLPs, where the first MLP is given sigmoid activation and the second MLP rescales the outputs. The second sub-network also has two MLPs in it, with tanh activation for the first MLP and second MLP rescales the outputs. This output is multiplied with goal directed unit vecto→r,− (), which is estimated by the approach described in Section 4.4. Finally, the two sub-networks outputs are aggregated by another MLP to giv→e− . Figure 2 (a) shows the net1 architecture. The net1 architecture can be expressed as: →− = (())→− () − ℎ(∆ ()) · (11) Here , , and are weight matrices. This net1 architecture mimics Equation 2. net2: Repulsive force from static obstacles This network is responsible for estimating the repulsive force from obstacles [24]. In this work, we only consider the nearest obstacle point. This force is defined by Equation 3. So, net2 architecture is designed according to this equation. Figure 2 (b) shows net2 architecture. The distance between pedestrian and obstacle B ( ), and the direction vector pointing towards B to →(− ) are network inputs. net2 consists of two MLPs, where the first one is followed by a sigmoid activation and another MLP rescales the outputs. net2 architecture can be expressed as: →− = ( /→ − ) · (12) Here, , and are weight matrices. This structure mimics SFMG force Equation 3. net3: Repulsive force from other pedestrians This section focuses on estimating the repulsive forces experienced by a pedestrian from other pedestrians. Equation 6 defines this force. So, this corresponding network aims to mimic this equation. The input parameters are →− , and→− . 1, 2, and are the learn-able parameters. Figure 2 (c) shows the network architecture used to estimate repulsive force exerted by one neighboring pedestrian. This network consists of two sub-networks. First sub-network take→s− and as inputs and it has one MLP which is followed by relu activation. This subsection estimates the first part of Equation 6. Second sub-network estimates second part of Equation 6 with inputs→:− , an→d− . It also has similar architecture as the first sub-network. Finally, outputs of these two sub-networks are combined by another MLP, to ge→t− force. The architecture is expressed as: →− = ((1 exp [ ()/→]− )+ (→2− →− →− () exp [− ()/ 2])) · (13) Here, 1, 2, , 2 and are weight matrices. This structure mimics Equation 6.
Again, one pedestrian can have multiple nearby pedestrian exerting repulsive force on it. So, we empirically consider a maximum of nine nearby pedestrians for a certain pedestrian. We employ instances of the network described above for finding repulsive forces from them. These networks are named as net3_i, where i = 1, 2, 3, ..., 9. A final network, net3 is used to combine the outputs of these networks to get total repulsive force from other pedestrians. net4: Group force net4 approximates the force exerted by pedestrian groups. The group force is given by Equation 7. Group force has two parts: visibility force () and attraction force (). So, net4 also has two sub-networks to estimate these two forces. Figure 2 (d) shows the net4 architecture. The first network inputs are an→d− . It has two MLPs: the first MLP is followed by relu activation, and second MLP rescales the relu output. The second sub-network has similar architecture, with→− ( , ) as input. The two outputs are combined by another MLP to estimate the final group force. This can be expressed as: →− = (relu(1→− ) + relu(→2− ( , ))) · (14) Here, 1, 2 and are weight matrices. This structure is similar to Equation 7. →− ()
This final recombination module aggregates the outputs of net1, net2, net3 and net4 to predict total force →− () . This network consists of two MLPs. First MLP is followed by relu activation, and second MLP rescales this output. Figure 2 shows the overall architecture. Expressed as: ⏟1⏞ ⏟ 2⏞ ⏟ 3⏞
⏟ 4⏞ = relu(→(− +→− + ∑︁→− +→− ) ) · (15) and are weight matrices. Note, as we consider only the nearest obstacle point, we do not need to sum repulsive forces from static obstacles.
As we compare, the overall architecture described by Equation 15 and portrayed in Figure 2, strongly emulates the complete SFMG force described by Equation 10.
An estimated goal or destination is required as a goal-directed unit vector→(− ) is an input for net1 and net3. In an environment with other pedestrians and static obstacles, a pedestrian’s previous positions cannot be exploited to estimat→e− . So, we use Multiple Model Approach (MMA) [29] to estimate it. Specifically, we use Interaction Multiple Model (IMM) estimators. Diferent goal hypotheses based Kalman filters are chosen as filters for the IMM estimator. The goal hypotheses were generated using the Constant turn rate and velocity (CTRV) [30] model. Then, the IMM estimator compares the diferent filter-generated trajectories with the observations and selects the most likely one.
In this section, we perform a set of experiments to qualitatively and quantitatively evaluate SFMGNet in terms of realistic (Section 5.4), interpretable (Section 5.5) and accurate (Section 5.6 and Section 5.7) trajectory prediction. We discuss the experimental setup, dataset details and feature extraction process in Sections 5.1, 5.2 and 5.3 respectively.
We train our model using a synthetic dataset to demonstrate our model’s performance in cases where the relevant data is scarce or unavailable. Only the models trained with synthetic dataset were used during the evaluation process, even while testing on real-world datasets. We create the synthetic dataset using the trajectories simulated by an SFMG (SFM extended with group force) based simulator 1 in a closed environment (crossing passageways). We run 1000 simulations of 30-second duration sampled at 0.1 seconds. The simulation run scenarios were chosen randomly 1SFM: https://github.com/svenkreiss/socialforce/tree/579543a3abe22716835acfa0b7d2d57fc1c199b6. Extended for groups. following a discrete uniform distribution. The following conditions were chosen to simulate scenarios rich with diferent types of interactions. The pedestrians’ starting positions were chosen randomly among four possible starting zones within 7 to 10 meters distance to the destination point. The number of pedestrians in a certain scenario was also chosen randomly between 2 to 10. The presence of a group of sizes 2 to 4 is chosen randomly. Relaxation time = 0.5, is constant. We divide the dataset with sample randomization into three sets: 50% for the training set, 25% for the development set, and 25% for the testing set.
We train the models net1, net2, net4, net3, and individual instances of net3 for diferent neighboring pedestrians (net3_1, net3_2, ..., net3_9) with an Adam optimizer with a learning rate of 0.001 and batch size 16. The number of epochs was chosen based on model performance on the development set. The final recombination layer was trained by Adam with a 0.1 learning rate. The window of motion observation is empirically chosen as n = 10. We train and optimize the model based on corresponding mean squared errors (MSE). The final MSE values for net1, net2, net3, net4 and SFMGNet are 0.0066, 0.5295, 0.1594, 0.0181 and 0.2640 units respectively.
To assess SFMGNet’s performance quantitatively, we test its performance on two benchmark datasets namely ETH [31] dataset and UCY [32] dataset. The ETH dataset consists of two scenes: ETH and Hotel. ETH scene consists of 360 pedestrians and 61 pedestrian groups. The hotel scene has 389 pedestrians and 41 groups. The UCY dataset has five scenes: namely zara01, zara02, students01, students03 and uni_examples. zara01 has 148 scenes and 45 groups. Zara02 consists of 204 pedestrians and 58 groups. students03 scene consists of 434 pedestrians and 104 groups. students01 consists of 414 pedestrians and uni_examples has 118 pedestrians. students01, students03, and uni_examples scenes together are called UNIV. All the datasets are sampled at 0.4 seconds. All these scenes consist of multiple pedestrians being present in the same frame. We use the group and destination-related information given by the work in [33].
For a certain pedestrian at time-step t, the SFMGNet model requires: distance and direction unit vector from the nearest obstacle ( an→d− ), distance and direction unit vector towards nearest (up to nine) pedestrian ( an→d− ). If in a group, then it needs a group Centroid directed unit vector→(− ( , )) and minimum rotation angle ( ). However, a trajectory dataset normally consists of time-step, velocity, and positions. Additionally, information about group formation [33] and obstacles are provided (or extracted). So, we follow the approach depicted in the flowchart in Figure 3 to extract features. A dataset (D), group (GroupList), and obstacle (ObstacleList) information are given to the algorithm. Then, at every time-step t in the dataset, we follow the following steps. First, the nearest obstacle is taken from ObstacleList, distance from pedestrian an→d− is calculated and stored in an array (ObstacleSet). Next, we check for multiple pedestrians’ presence. If true, we treat as neighbors to ego pedestrian ( ) and, calculate and store and→− in NeighbourList. Next, we check the existence of groups from GroupList. If true, first we get the corresponding group member positions. Now, we nee→d− ( , ) and . For calculatin→g− ( , ), first, we find the group Centroid based on Start
Dataset, GroupList,
ObstacleList
Yes Get distance and direction to nearest obstacle
present? positions No
ThetaSet
Find ? etaCentroidSet
Find direction vector towards cetroid choose leader randomly Find Group centroid
Get corresponding distance and direction vector
neighbourSet No
Get member YesIn GroupList? positions group member positions and find their corresponding→− ( , ). Next, we store it in an array (etaCentroidList). For , first, we randomly choose a group leader, then we calculate based on the group leader and other member positions (see [4], [27]), and store it in an array (ThetaSet).
This section aims to evaluate SFMGNet’s ability to predict realistic trajectories through a case study. For this, we choose a sample scenario (time-step: 4630 - 4880) from the ETH scene from ETH dataset [31] with multiple numbers of pedestrians and pedestrian groups. This scenario consists of eight pedestrians namely: 84, 85, 86, 87, 88, 89, 90 and 91. Here, 86, 87, and 91 are traveling from right to left direction. And other pedestrians are traveling in the opposite direction. 84 and 85 are in a pedestrian group. Similarly, 88 and 89 are also in a group. We 0 0 4 4 8 8 predicted force
Simulated force 12 16
20 predicted force
Simulated force 12 16 20 10 0 8 )m6 ( s i x -Ya 4 2 0 4 8
12 #85 SFMGNet #85 real #85 SFMG 16
20 0 2 4 6 X-axis (m) 8
10 predict all the pedestrian trajectories with SFMGNet by taking 1.2 seconds of previous motion information as input and predicting the next 4.8 seconds of the trajectory (if possible) at a time. We also simulate their trajectories with SFMG [3] [4] for comparison. Figure 4 shows the real (solid lines), predicted (dashed lines) and SFMG simulated (dots) trajectories. Arrows show the corresponding pedestrian’s motion direction. We see that predicted trajectories are similar to the real pedestrian trajectories, barring a couple of time steps. The SFMGNet can predict and model slight deviations in every pedestrian’s motion accurately. Conversely, SFMG simulated trajectories are flatter, not covering the influences of environment related influences on the pedestrian. Also, SFMG simulated pedestrian trajectories fail to reach or overtake the destination point. As an example, we take a look at 85 in Figure 5 (a). The numbers represent the time steps. Our model predicted trajectory matches with the real trajectory and reaches the destination. Whereas, SFMG simulated trajectory is flatter and fails to reach the destination.
In this section, we aim to interpret the SFMGNet predicted trajectories to a certain degree as a proof of concept. For that, we look at the predicted and simulated forces acting upon a certain pedestrian and try to establish a causal relationship with the environment elements/influences. We choose pedestrian 85 for this evaluation. Figure 4 shows the overall scenario, Figure 5 zooms into 85 trajectories (a) and, predicted and SFMG simulated total force (b). Looking into the predicted total force (Figure 5 (b)), we see an almost constant force acting upon the pedestrian, approximately 0.01 units on X-axis and 0.01 units on Y-axis. It points to the idea, that the pedestrian maintains a rather stable trajectory with subtle direction changes in its journey. The SFMG simulated force also remains stable apart from a spike at the beginning. However, the total force does not paint the whole picture. So, we explore the individual forces.
Figure 6 shows the individual forces, namely: acceleration force (a), the repulsive force from boundaries (b), repulsive from other pedestrians (c), and group force (d). The predicted attraction force (Figure 6 (a)) changes over time, starting at positive values, falling towards zero at around time-step 12, and continues falling afterward. This could point toward the pedestrian’s need to change its velocity overtime to compensate for other environmental facts. That is, it would feel like a constant acceleration force in an open environment with no other road users and obstacles. But it’s not the case here. However, the SFMG simulated acceleration force remains stable throughout. Next, we look at the forces exerted by other actors; static and dynamic. Figure 6 (b) shows that, the pedestrian feels a constant repulsive force (both predicted and simulated) from boundaries throughout its journey. This corresponds to its desire to keep a safe distance from boundaries. As the nearest boundary point is quite far from it, the force values are low. Again, the pedestrian is surrounded by seven other pedestrians, so, it should feel a strong cumulative repulsive force from other pedestrians. This behavior is emulated by the predicted repulsive force from other pedestrians shown in Figure 6 (c). However, SFMG fails to emulate this properly, as it simulates force values nearing zero units. Moreover, 85 is traveling in a group with 84. Therefore, it feels like group forces influencing it, as we see in Figure 6 (d). The SFMGNet predicted group forces are much higher (in magnitude) compared to SFMG simulated group forces. The predicted group force better represents 85’s desire to stay closer to 84. The nature of group force depends on the combination of the visibility force, and attraction force towards group centroid, . But, we do not consider that level of granularity in this work to avoid further complexity. To conclude, we can establish a high-level causality behind a pedestrian’s motion behavior motivation by inspecting individual force predictions.
We quantitatively assess our model based on two commonly used [13], [28] distance-based parameters: average displacement error (ADE) and mean final displacement error (FDE). ADE refers to the mean of all euclidean distance values between the real and predicted values at each time-steps. FDE is the distance between the last predicted position and the last real position. The lower the values of both ADE and FDE parameters, the better the model performance.
For comparison, we choose constant velocity model (CV), a MLP or feed-forward networkbased model [11] (FF), S-LSTM model [13], S-GAN model [16] and the model described in [24] (SFM-NN) as baselines. The baseline models (expect SFM-NN) take 3.2 seconds (8 time-steps) of trajectory values as input and predict the next 4.8 seconds of trajectory, and report error values. They are also trained on the corresponding real-world datasets. The SFM-NN is trained on a synthetic dataset and takes 1 second of data as input to predict 4.8 seconds of trajectory. Our model (SFMGNet) is trained solely on a synthetic dataset. It takes 1.2 seconds (as n = 10 chosen) of data as input and predicts the next 4.8 seconds of data. Then, we calculate the error values. The error values are reported in Table 1. The bold values represent the lowest or best error values. As we can see, our model SFMGNet outperforms every baseline model in every scene dataset. Especially, the lowest ADE and FDE values (i.e. 0.07 and 0.11 respectively) are found for hotel scene data in ETH dataset. Our model shows the worst performance on UNIV dataset: ADE = 0.35 and FDE = 0.64. Still, the model performance is noticeably better than the baseline models. The average ADE and FDE values are: 0.15 and 0.26.
We evaluate SFMGNet’s ability to predict feasible and realistic trajectories by calculating the evaluation metric introduced in [34]. That is, we calculate the percentage of near-collisions among pedestrians for each frame or time-step. A near-collision takes place when the Euclidean distance between the pedestrians is below 0.1 meters [34]. We compute the average percentage of human near-collision in each frame of ETH and UCY datasets. Our model (SFMGNet) predicted trajectories do not contain any near-collisions in both datasets. That is, SFMGNet predicts 0.0% near-collisions for all frames in ETH, ETH hotel, UNIV, zara01, and zara02 scenes. This in turn strongly indicates that SFMGNet can predict feasible and realistic trajectories.
To assess the impact of diferent modules in our proposed model framework, i.e. acceleration force, the repulsive force from static obstacles, the repulsive force from other pedestrians, and group force, we test the performance of several ablative models. We choose three diferent model versions, namely: Att, ABr, and ABrPr for this study. Where the model Att considers only acceleration force towards the destination, model ABr considers both acceleration force and repulsive force from static obstacles, and model ABrPr considers all forces except group force. Finally, we compare them with the complete framework, SFMGNet. We test them on ETH and UCY datasets and report the corresponding ADE/FDE error values in Table 2. We can see, the model performance improves with the addition of each force estimation module. Model Att performs the worst across all datasets, and performance increases significantly when repulsive force from obstacles is considered in model ABr. This trend follows for model ABrPr, which considers repulsive force from other pedestrians. Models Att, ABr, and ABrPr perform notably worse on the ETH scene compared to the other scenes. However, SFMGNet outperforms these ablative models significantly, indicating the strong importance of pedestrian groups.
Based on the above quantitative and qualitative evaluation, we conclude that SFMGNet can model realistic, interpretable pedestrian trajectories. Also, it shows better than state-of-the-art performance on two real-world benchmark datasets in terms of distance-based metrics.
This work proposes SFMGNet, a physics-based neural network-based framework to predict pedestrian trajectories while considering the influences of static obstacles, other pedestrians, and pedestrian groups. SFMGNet combines the social force model with group behavior modeling (SFMG) and multi-layer perceptron (MLP). SFMG provides knowledge about the motivation behind pedestrian motion behavior, which is then engineered in the MLP based network’s architecture. This combination allows the MLP based model to inherit the SFMG model’s interpretable nature. This model has the potential to contribute to better planning and understanding motion behavior of robots and autonomous vehicles. Based on the evaluation results, we conclude that the model predicts realistic trajectories and outperforms state-of-the-art models on the ETH and UCY datasets. We can also causally interpret model predictions. However, there remains potential for further improvement. We aim to include other types of road users (e.g., cars, cyclists) in the model to better emulate heterogeneous mixed-trafic zones. Again, we want to explore our model’s potential in modeling new types of road users (e.g., cargo-movers, tram buses), even when relevant data is scarce/absent. [1] A. Talebpour, H. S. Mahmassani, Influence of connected and autonomous vehicles on trafic lfow stability and throughput, Transportation Research Part C: Emerging Technologies 71 (2016) 143–163. [2] P. Bhavsar, P. Das, M. Paugh, K. Dey, M. Chowdhury, Risk analysis of autonomous vehicles in mixed trafic streams, Transportation Research Record 2625 (2017) 51–61. [3] D. Helbing, P. Molnar, Social force model for pedestrian dynamics, Physical review E 51 (1995) 4282. [4] S. Ahmed, F. T. Johora, J. P. Müller, Investigating the role of pedestrian groups in shared spaces through simulation modeling, in: International Workshop on Simulation Science, Springer, 2019, pp. 52–69. [5] S. Hossain, Predicting pedestrian motion using a physics-based neural network, Master’s thesis, Clausthal University of Technology, 2021. [6] F. Pascucci, N. Rinke, C. Schiermeyer, V. Berkhahn, B. Friedrich, Should I Stay or Should I Go? A Discrete Choice Model for Pedestrian–Vehicle Conflicts in Shared Space, Technical Report, 2018. [7] N. Rinke, C. Schiermeyer, F. Pascucci, V. Berkhahn, B. Friedrich, A multi-layer social force approach to model interactions in shared spaces using collision prediction, Transportation Research Procedia 25 (2017) 25 (2017) 1249–1267. [8] B. Anvari, M. G. Bell, A. Sivakumar, W. Y. Ochieng, Modelling shared space users via rule-based social force model, Transportation Research Part C: Emerging Technologies 51 (2015) 83–103. [9] F. T. Johora, J. P. Müller, Zone-specific interaction modeling of pedestrians and cars in shared spaces, Transportation Research Procedia 47 (2020) 251–258. [10] H. Cheng, F. T. Johora, M. Sester, J. P. Müller, Trajectory modelling in shared spaces: Expert-based vs. deep learning approach, in: 21st International Workshop on MultiAgent Systems and Agent-Based Simulation (MABS 2020), Auckland, NZ, 2020. URL: https://samarthswarup.github.io/mabs2020/accepted/. [11] C. Schöller, V. Aravantinos, F. Lay, A. Knoll, What the constant velocity model can teach us about pedestrian motion prediction, IEEE Robotics and Automation Letters 5 (2020) 1696–1703. [12] X. Wang, R. Jiang, L. Li, Y. Lin, X. Zheng, F.-Y. Wang, Capturing car-following behaviors by deep learning, IEEE Transactions on Intelligent Transportation Systems 19 (2017) 910–920. [13] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, S. Savarese, Social lstm: Human trajectory prediction in crowded spaces, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 961–971. [14] H. Cheng, M. Sester, Modeling mixed trafic in shared space using lstm with probability density mapping, in: 21st International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2018, pp. 3898–3904. [15] H. Cheng, W. Liao, M. Y. Yang, B. Rosenhahn, M. Sester, Amenet: Attentive maps encoder network for trajectory prediction, arXiv preprint arXiv:2006.08264 (2020). [16] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, A. Alahi, Social gan: Socially acceptable trajectories with generative adversarial networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2255–2264. [17] A. Vemula, K. Muelling, J. Oh, Social attention: Modeling attention in human crowds, in: 2018 IEEE international Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 4601–4607. [18] Y. Huang, H. Bi, Z. Li, T. Mao, Z. Wang, Stgat: Modeling spatial-temporal interactions for human trajectory prediction, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6272–6281. [19] F. Pasquale, Toward a fourth law of robotics: Preserving attribution, responsibility, and explainability in an algorithmic society, Ohio St. LJ 78 (2017) 1243. [20] A. Kroll, Grey-box models: Concepts and application, New frontiers in computational intelligence and its applications 57 (2000) 42–51. [21] F. T. Johora, H. Cheng, J. P. Müller, M. Sester, An agent-based model for trajectory modelling in shared spaces: A combination of expert-based and deep learning approaches, in: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 2020, pp. 1878–1880. [22] B. S. Kim, T. G. Kim, Modeling and simulation using artificial neural network-embedded cellular automata, IEEE Access 8 (2020) 24056–24061. [23] A. Karpatne, G. Atluri, J. H. Faghmous, M. Steinbach, A. Banerjee, A. Ganguly, S. Shekhar, N. Samatova, V. Kumar, Theory-guided data science: A new paradigm for scientific discovery from data, IEEE Transactions on knowledge and data engineering 29 (2017) 2318–2331. [24] A. Antonucci, G. P. R. Papini, L. Palopoli, D. Fontanelli, Generating reliable and eficient predictions of human motion: A promising encounter between physics and neural networks, arXiv preprint arXiv:2006.08429 (2020). [25] S. Kreiss, Deep social force, arXiv preprint arXiv:2109.12081 (2021). [26] M. Moussaïd, N. Perozo, S. Garnier, D. Helbing, G. Theraulaz, The walking behaviour of pedestrian social groups and its impact on crowd dynamics, PloS one 5 (2010) e10047. [27] A. Kremyzas, N. Jaklin, R. Geraerts, Towards social behavior in virtual-agent navigation,
Science China Information Sciences 59 (2016) 1–17. [28] H. Sun, Z. Zhao, Z. He, Reciprocal learning networks for human trajectory prediction, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7416–7425. [29] Y. Bar-Shalom, X. R. Li, T. Kirubarajan, Estimation with applications to tracking and navigation: theory algorithms and software, John Wiley & Sons, 2004. [30] S. Blackman, R. Popoli, Design and analysis of modern tracking systems(book), Norwood,
MA: Artech House, 1999. (1999). [31] S. Pellegrini, A. Ess, K. Schindler, L. Van Gool, You’ll never walk alone: Modeling social behavior for multi-target tracking, in: 2009 IEEE 12th International Conference on Computer Vision, IEEE, 2009, pp. 261–268. [32] A. Lerner, Y. Chrysanthou, D. Lischinski, Crowds by example, in: Computer graphics forum, volume 26, Wiley Online Library, 2007, pp. 655–664. [33] J. Amirian, B. Zhang, F. V. Castro, J. J. Baldelomar, J.-B. Hayet, J. Pettre, Opentraj: Assessing prediction complexity in human trajectories datasets, in: Asian Conference on Computer Vision (ACCV), CONF, Springer, 2020. [34] A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, S. Savarese, Sophie: An attentive gan for predicting paths compliant to social and physical constraints, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 1349–1358. doi:10.1109/CVPR.2019.00144.