It is a fact that the world population has grown in recent decades, as well as the number of social and tourism events, generating situations of agglomerations where diferent problems may lead to generate bottlenecks stampedes or falls, that can be a risk for people. Thus, the study of the behaviour of crowds is a relevant research topic. In this context, this paper presents and approach for real-time stampede detection from images, in low and medium crowd scenarios. The proposal is based on a feature vector extracted from the optical flow entropy, and this does not require the use of thresholds. Instead of that, it includes a a Stacking classifier, based on the union of a random forest with ten estimators and an support vector classifier, that works properly in the diferent analyzed scenarios. The proposal has been evaluated in UMN and PETS 2009 datasets and compared to other state-of-the-art proposals in terms of accuracy and computational cost. However, since the provided ground-truth was not accurate, a new manually-labelled ground-truth has been generated and make publicly available to the scientific community. The obtained results allows validating the proposal, outperforming the state-of-the-art methods both in terms of accuracy and computational cost in all the evaluated scenarios.
Crowd analysis, in general, can be approached holistically or by object-based methods. object
based methods analyse crowds as sets of objects, studying people in a particular way [
In this work, we have focused on the detection of anomalous behavior in crowds. In particular,
in the behaviour related to stampedes. Detection systems usually have two diferentiated phases.
The first phase is based on the representation of the event by a set of characteristics. There
are many methods for the extraction of features, such as the study of social force [
The latest research in the detection of abnormal behaviour in crowds uses technologies such as context location and motion-rich patio-temporal volumes [9],temporal convolutional neural network pattern [10], generative adversarial networks [11], global event influence model [ 12]. In this work, we have implemented a system that draws from both the most modern and the most classical approaches, Being a system based on the use of the value of the magnitude of the optical flow and from there extract the entropy to generate a series of descriptors that are used in a machine learning model for the detection of the anomaly.
Considering the previous research work, the main contribution of this paper is to deploy a
robust and reliable system capable to detect stampedes in real-time. In addition, the algorithm
to detect events’ peak does not require a threshold. Additionally, we have manually labelled the
UMN [
The rest of the paper is organized as follows: section 2 describes the proposal for stampede detection, them section 3 presents the annotation procedure for the UMN and PETS-2009 datasets. Next, the main experimental results are shown in section 4.1. Finally, section 5 describes the main conclusions and future work.
We have focused our eforts on identifying stampedes in the first two types of scenarios, the low and intermediate density crowds. As other previous approaches, such as the one described in [15], the proposal presented in this work is based on analysing the entropy obtained from the scene optical flow. It is obtained a dense optical flow by using the method of Farneback [ 16], instead of a punctual one as in the case of using the method of Lucas-Kanade [17].
Besides, the system works in real time. Moreover, it does not require a threshold (that must be modified for each dataset). Instead, we have extracted a feature vector from the entropy, and designed a machine learning model based on stacking classifier for stampede detection that uses a set of features generated from the entropy signal extracted from the optical flow.
The figure 1 shows a general block diagram with each of the stages of the system. Below, there are described in detail each of these stages.
The system extracts the entropy using the optical flow of the image. For this purpose, we have used the dense optical flow of Farneback instead of the point optical flow of Lucas-Kanade. Because we wanted a system that would analyse the whole image.This system obtains the movement variations of all pixels between frames. Unlike Lucas-Kanade, which is based on the study of the variations of a specific set of pixels.
The Farneback’s method is based on the estimation of the movement of the pixels between the actual and previous frames. From that movement, there are generated the displacement vectors, that are then used to study the movement variations in the image. This type of optical lfow analysis presents a higher accuracy than those algorithms based on sparse optical flow.
Thus, Farneback’s dense optical flow extraction is based on expanding the position of the pixel coordinates by polynomial expansion using the neighbourhood information of each pixel in the image. The original coordinates ( 0, 0) are independent variables, and the new coordinates (, ) are polynomials of dependent variables. The amount of motion (, ) of the pixel in and directions are determined by substituting the coordinates into them. A displacement vector is obtained for each pixel between two frames.
The feature vector used for detecting stampedes is based on the entropy. To obtain the entropy, ifrst, the image is pre-processed by blurring and grayscaling. Then the magnitude value of the optical flow is extracted using the current and previous frames These magnitude values form a matrix that correspond to the movement variation of each pixel. Then, these matrices are grouped in batches of 20 frames, and the mean of all the magnitudes is extracted for each pixel. Then the median is calculated, and the result is called activity map. From the activity map, the entropy for that frame can be extracted. To extract the entropy it is used equation 1, where is the number of separate symbols, is the frequency of the each pixel in the image and is the actual frame.
() = ∑ =1 2( )
In previous works, such as [15], the authors use two thresholds to determine if there was a stampede or not. The first one was based on the entropy value, and the second one was based on the temporal occupancy variation (TOV) value between frames. For this work, the TOV has not been employed using only the entropy value, from what there is obtained a feature vector for the stacking classifier.
After obtaining the entropy, to determine if there is a stampede or not, we extract a set of features that are next classified using a stacking classifier. These features include the mean and standard deviation of the entropy. The mean is computed using a sliding window of 20 frames as shown in the equation 2, being the size of the sliding windows . Thus the proposal requires a minimum of 20 frames for detecting a stampede, but it smooths the signal and removes the (1) (2) (3) (4)
The standard deviation of the entropy descriptor is used to detect the fast changes in the signal. A considerable variation generates a significant change of value in this descriptor. The equation 3 shows the mathematical definition used for the extraction of the descriptor, where are the current entropy values and the mean entropy value.
() = √
∑ =−+1 ( () − ()) 2
The third feature is the distance generated as the diference between the mean plus standard deviation and the mean minus standard deviation. The standard deviation is multiplied because this value ( ) is smaller than the other signal. In the equation 4 is defined mathematically. high frequency noise.
() = 1 =−+1 ∑
() () = (() + ∗ ()) − (() − ∗ ())
An example of the signals explained below are shown in the figure 2. The peak of the entropy (right signal) reflects the moment when the stampede happens. The features explained below are drawn in the left picture. In blue, it is shown the entropy mean in yellow, the standard deviation and in green with shade the distance.
The stacking classifier model (figure 3) has been used, which is a combination of an support vector classifier (SVC), random forest (RF). It is an ensemble learning technique for combining multiple classification models through a meta-classifier. The individual classification models are trained based on the complete training set; then, the meta-classifier is tuned based on the results-meta-features of the individual classification models in the ensemble. The meta-classifier can be trained on the predicted class labels or the ensemble probabilities. Figure 3 shows the basic structure of a stacking classifier.
The stacking classifier used in this work is based on the union of a random forest with ten estimators and an SVC, that has been used to classify if there is a stampede or not. The training was performed with half of the UMN videos and half of the PETS videos, being the other half used for model testing. This 50:50 margin is used for two reasons: first, a stacking classifier does not need a large amount of information to train and second, the number of cases in which there is a stampede in the videos is much smaller than in those in which there is not, so more videos have been used for testing to achieve a more reliable result. In order to obtain the best results in terms of measuring the predictive quality of the models, a k-fold of the training process has been performed with a value of k equal to 10. The value of is 10, because the datasets have a small number of frames and a small number of labels, so the division into 10 slots provides enough information in each slot for the training to succeed.
To evaluate the system, we have used two datasets, UMN and PETS2009, that have been widely used in other works for stampede detection. These datasets include several videos with stampedes. The UMN dataset include the ground-truth, however, when analysing the videos, it can be seen that there is a delay between the beginning of the stampede and the frame in which it is labelled. Furthermore, PETS dataset does not include ground-truth information for stampede events.
The characteristics of the datasets are shown in the table 1. Both datasets have similar types of stampedes, in which people run either in the same direction or spread out. The way of recording the videos is the same for both datasets by high-angle shot. The lighting is constant in all videos but one environment of the UMN dataset. The big diference between the two datasets is the number of people in UMN is not more than 20 people in any video, while in PETS there are more than 30 people per video.
Due to the lack of an accurate ground-truth, we have analysed the two dataset, and handlabelled the information. To label the videos, it is necessary to define two moments in a stampede, the beginning and the end of the stampede. To label the initial moment we have considered that more than four people in the image are already prepared to run or moving. The end of the stampede is defined when the people in the image stop running and start walking or when less than three people are running on the screen. By means of these guidelines, we have defined a ground-truth made by hand analysing frame by frame and indicating the moment when each stampede starts and ends. Figure 4 shows a ground-truth scheme indicating in green the moments of calm and in red the times where it is considered that there is a stampede. In addition, in Figure 4, the frames where the stampede begins and ends and the total number of frames of each video are shown.
To evaluate the system performance, we have analysed two key parameters. The first one is the speed of execution of the system, and the second one the accuracy to detect stampedes.
The computational cost has been evaluated for the original system executed on a CPU (Intel Core i7-9700K CPU @ 3.60GHz x 8). It is important to note that the OpenCV version is 4.5.1, and the programming language is Python version 3.7.
Regarding the accuracy, as it has been stated before, two datasets (UMN and PETS) have been used, which have been compared with the results obtained in the work [15] . The UMN dataset includes a total of 7502 frames with a resolution of 240x320 at 30 fps. The dataset has been divided into three scenarios, two outdoor and one indoor. The PETS dataset includes a single outdoor scenario with a total of 2872 frames at a resolution of 768x576 at 30 fps.
To evaluate the system we have decided to use the receiver operating characteristic (ROC) and area under the curve (AUC) metrics. A ROC curve is a graph showing the performance of a classification model at all classification thresholds. This curve represents the true positives rate (TPR) versus false positives rate (FPR) at diferent classification thresholds, whereas the AUC measures the entire two-dimensional area below the total ROC curve. The AUC provides an aggregate measure of performance for all possible classification thresholds.
This section shows the results of stampede detection in diferent environments, both indoors and outdoors corresponding to the previously described UMN and PETS 2009 datasets.
As explained previously, entropy, mean, standard deviation and distance combination among entropy, mean and standard deviation are extracted of each frame. These values are analysed by a stacking classifier that determine the value of the activity.
Figure 6 shows an example of UMN where it can be clearly observed where the stampede starts, marked with an arrow.
Due to the frame characteristics of each dataset, it is necessary to normalise the magnitude values from the optical flow. This normalisation is performed so that the entropy values are limited between two close values. This allows the dispersion between the diferent samples to be smaller and helps the classifier to detect better. This normalisation is adjusted to the number of pixels in the image by dividing the magnitude given by the optical flow by a constant that will be larger as the image size increases. Figure 7 shows an example in which the maximum value of the entropy is similar to the maximum value in UMN (figure 7).
To observe the improvement of the system performance, we compare the ROC curves of our system with those provided in the baseline [15]. It is worth note that the ground-truth used by this paper [15] defines the beginning and the end of the stampede too late in relation to the actual times of the videos. For this reason we have manually-labelled the videos, using the above mentioned definition of stampede. Thus, our ground-truth is more accurate and realistic than the one used by [15]
The first scenario is the UMN lawn (figure 8), in which no new actors enter the scene. Figure 8a shows an example of a frame corresponding to this scenario, whereas figure 8b compares the ROCs and the AUC for our system (in red) and for the baseline [15] (in blue). Comparing the ROCs, it can be seen that the results are similar, although some improvement is obtained in our system.
(a) Lawn scenario. (b) ROC lawn scenario: red our system and in blue the system [15]
The indoor scenario (figure 9) is characterized for being a recording of a hall, where unusual activity is observed, such as the entrance and exit of people, as well as much more abrupt light changes than in the other two scenarios. The indoor scenario is where the most significant improvement is observed, having important increase in the area under the curve (AUC), being the colour red our system and in the colour blue, the [15] system (figure 9b).
(a) Indoor scenario. (b) ROC indoor scenario: red our system and in blue the system [15]
Finally, figure 10 shows the last scenario of the UMN dataset. It is recorded in a plaza, where the people is walked. This scenario (figure 10a) has the lighting, the size of the people, recording environment similar to the lawn scenario. For this reason, the values obtained in the ROC curve (figure 10b) are practically equal to the lawn scenario. As it is shown in the figure the AUC obtained by our system (colour red) and the system [15] (colour blue) is very similar, having minor improvements.
(a) Plaza scenario.
Regarding the PETS 2009 dataset (figure 11), we find a view similar to the lawn and plaza scenaios in the UMN dataset. The frames show a street crossing where a group of people run in a stampede (figure 11a). The AUC obtained with our method provides a better operation than [15].
As it has been explained before, our proposal is able to perform in real time, being faster (b) ROC plaza scenario: red our system and in
blue the system [15] (a) PETS scenario. (b) ROC PETS 2009: red our system and in blue the system [15] than [15]. In the table 2 is shown a comparison between our model and the [15] model. In the table there are compared both frame per second processed by each proposal and the AUC value. It can be seen that our model obtains better results in every field, both in computational cost (more than 2x speed up) and AUC (with and improvement from +3% to +10%, depending on the dataset).
In this work, we have proposed an approach for real-time stampede detection in low and medium density crowd scenarios. The proposal is based on a feature vector extracted from the optical flow entropy, and this does not require the use of thresholds, since there has been replaced by a stacking classifier, based on the union of a random forest with ten estimators and an SVC, that works properly in the diferent analysed scenarios.
We have selected and extracted features for stampede detection. These features are extracted through the entropy information given by the optical flow. These features are suitable descriptors for the detection of the stampede beginning.
The proposal has been evaluated exhaustively in UMN and PETS 2009 datasets, that has been widely used in stampede detection works, and compared to other state-of-the-art proposals in terms of accuracy and computational cost. For this evaluation, the UMN and PETS (stampedes) datasets have been hand labelled, adjusting in a more precise way the start and end frames of the stampedes. For this reason, our ground-truth is tighter and more precise than the existing ones.
As conclusion, the system ofers an improvement of the results respect to previous state-ofthe-art works both in precision and computational cost. In addition the system does not depends on a threshold to run correctly and it is able to adapt to diferent environments. This has been possible by replacing thresholds with machine learning models that are able to generalise a solution independently of the environment.
As future lines of work, we plan to develop this system but running on dedicated vision processing hardware such as GPU or VPU. This will allow a faster response in the processing and add another layer of parallelisation to the system. Regarding the classification method, it is proposed to use a dense neural network instead of a classical machine learning model and compare how the system behaves in deep learning models. In addition, it is intended to test the system on more datasets in order to evaluate more environments. Evaluating in a more extensive way the generality of the designed system.
The authors would like to thank the GEINTRA research group (geintra-uah.org) for their support and background work. This research has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under PALAEMON project (Grant Agreement nº 814962), and by the Spanish Ministry of Economy and Competitiveness under project HEIMDALUAH (TIN2016-75982-C2-1-R). [7] S. Hawkins, H. He, G. Williams, R. Baxter, Outlier detection using replicator neural networks, in: International Conference on Data Warehousing and Knowledge Discovery, Springer, 2002, p. 170–180. [8] S. Zhou, W. Shen, D. Zeng, M. Fang, Y. Wei, Z. Zhang, Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes, Signal Processing: Image Communication 47 (2016) 358–368. [9] N. Patil, P. K. Biswas, Global abnormal events detection in crowded scenes using context location and motion-rich spatio-temporal volumes, IET Image Processing 12 (2018) 596–604. [10] M. Ravanbakhsh, M. Nabi, H. Mousavi, E. Sangineto, N. Sebe, Plug-and-play cnn for crowd motion analysis: An application in abnormal event detection, in: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2018, p. 1689–1698. [11] M. Ravanbakhsh, M. Nabi, E. Sangineto, L. Marcenaro, C. Regazzoni, N. Sebe, Abnormal event detection in videos using generative adversarial nets, in: 2017 IEEE International Conference on Image Processing (ICIP), IEEE, 2017, p. 1577–1581. [12] L. Pan, H. Zhou, Y. Liu, M. Wang, Global event influence model: integrating crowd motion and social psychology for global anomaly detection in dense crowds, Journal of Electronic Imaging 28 (2019) 023033. [13] J. Ferryman, A. Shahrokni, Pets2009: Dataset and challenge, in: 2009 Twelfth IEEE international workshop on performance evaluation of tracking and surveillance, IEEE, 2009, p. 1–6. [14] A. C. Cob-Parro, Umn repository with the updated grountruth, 2021.
Https://github.com/CarlosCobParro/UMN-groundtruth-update. [15] A. Pennisi, D. D. Bloisi, L. Iocchi, Online real-time crowd behavior detection in video sequences, Computer Vision and Image Understanding 144 (2016) 166–176. [16] G. Farnebäck, Two-frame motion estimation based on polynomial expansion, in: Scandinavian conference on Image analysis, Springer, 2003, p. 363–370. [17] B. D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in: Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’81, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1981, p. 674–679.