Falls in people are frequent and dangerous problems that can lead to death or infirmity, especially as people age. Therefore, fall detection and immediate intervention is essential to prevent more severe consequences. In this paper, two Fall Detection systems have been proposed. The proposed systems use 2D and 3D pose estimation of subjects respectively and a LSTM deep neural network architecture. An Attention Mechanism has been implemented in the neural network architecture to improve the performance of fall detection. The conducted experiments show that the model created with attention can generalize and does not suffer from scene bias. The 3D system using the Attention Mechanism has achieved an accuracy of 82% compared to the accuracy of 70% obtained without attention. With the 2D system using the neural network with Attention an accuracy of 72% has been obtained, instead, without attention, an accuracy of 52% has been obtained.
Falling in the elderly is a public health problem as it can cause disabling fractures and, in addition,
can also lead to psychological consequences that reduce a person's independence. Approximately
28-35% of people aged 65 years and older fall each year [
In this paper, two Fall Detection systems are proposed. The first system uses 2D poseestimation of subjects, and the second uses 3D pose-estimation of subjects. The neural network used for both systems mainly include LSTM and Attention Mechanism. Through this study, it is possible to show the efficiency of the Attention Mechanism in promptly detecting fall patterns from time series of features extracted from body joint coordinates.
The paper is structured as follows: the second section illustrates the state of the art of algorithms used by applications created for this purpose. Section three illustrates the Datasets used and the design of our experiment. Then, in Section four, experimental results will be described. Finally, Section five describes the conclusions and planned future work. ____________________________________________ WAMWB’23: Workshop on Advances of Mobile and Wearable Biometrics, September 26, 2023, Athens, Greece francesco.castro@unicampus.it (F. Castro); vincenzo.dentamaro@uniba.it (V. Dentamaro); vincenzo.gattulli@uniba.it (V. Gattulli); donato.impedovo@uniba.it (D. Impedovo)
0000-0002-8579-8941 (F. Castro); 0000-0003-1148-332X (V. Dentamaro); 0000-0001-9974-9414 (V. Gattulli); 0000-0002-9285-2555 (D. Impedovo) © 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
In Fall Detection, the main techniques used can be divided into three categories: 1. Based on wearable devices; 2. Based on environmental sensors; 3. Based on image processing.
The architecture of systems based on wearable devices and environmental sensors is presented in Fig. 1:
In contrast, the architecture of vision-based systems i.e., using video or cameras is shown in
Fig. 2 as discussed in [
Fall detection techniques based on wearable devices (smartwatches, pendants, etc...) have
built-in sensors that can detect the movement and position of the subject wearing them. One of
the most widely used methods to measure physical activities using wearable devices is the use of
the accelerometer, which involves measuring the acceleration of various body parts in time.
However, a barometric pressure sensor was introduced in [
Physiological responses such as changes in heart rate or blood pressure can result from
physical activity and changes in body position, and these physiological responses can be used for
fall detection. In the paper [
Environment-based Fall Detection systems can monitor human posture to detect falls. Systems
based on wearable sensors, as opposed to environment-based systems, are not sensitive to
environmental changes because they do not consider environmental factors. However, ecological
systems provide a solution by collecting data from the user and examining the environment.
These systems use external sensors installed around the area where the user's daily activities are
carried out, such as a home or elderly care facility [
Overall, these environmental systems have the advantage of operating in a low-light region and not being limited by privacy issues compared to vision-based systems. These systems also provide a more comprehensive analysis of the user's posture by considering the environmental factors than wearable systems based on inertial sensors. The limitations of the ecological system are as follows: • They are only suitable for indoor environments and cannot be installed or used outdoors; • Since the sensors are always in fixed positions, it is challenging to implement such systems; • •
Most environmental systems can only detect one person in the monitored area, which means no pets, no partners, and no friends; Although the sensors used are inexpensive, installing such a system requires a major renovation of the house because most of the sensors will be embedded under the floor or in the wall, which can be an expensive setup.
In addition to wearable devices and environmental sensors, vision-based systems using
cameras can also be used for fall detection, which requires immediate assistance if a fall occurs.
These systems usually use depth cameras or RGB cameras; depth cameras can calculate 3D
information using a single camera and perform better in low light conditions [
Fall detection systems based on image processing are like environmental systems in that they
share advantages in analyzing ecological factors, which wearable devices do not have; plus, both
have the problem that they can only be used indoors and often have blind spots and huge
expenses. However, image processing systems have unique advantages. Thanks to pattern
recognition, image processing systems can identify, track, and monitor the target user even when
multiple people or pets are in the monitored area. However, such video recording systems are
always subject to privacy issues, plus by using complex real-time image processing algorithms,
they require a huge amount of space and computing power [
Two datasets were used for the realization of the Fall Detection system. The first dataset is the UR Fall Detection Dataset (URFD). It contains: • Total of 70 videos • These videos have both images in RGB format and another format, but only those in RGB format were considered in our case. • 30 videos contain falls and 40 do not. • 30 videos containing falls do not directly contain the person falling, but rather, the person performing daily activities falls in different places. This way, an attempt is made to make the fall as realistic as possible. • 40 videos that do not contain falls present people performing these daily activities in different places, such as lying down, sitting, etc.; • There are also fall-like movements, such as squatting and picking up an object from the ground, to confuse future classification.
The second dataset used Le2i is the one described in (Charfi et al., 2012); It contains: • 250 videos, of which 192 contain falls and 57 contain everyday activities. • The videos show the main difficulties an elderly person might encounter in an enclosed environment. • In the case of videos with falls, as in the previous dataset, we have a person while performing daily activities fall; on the other hand, in the videos where falling is not present also here people perform various activities such as lying down, picking up an object from the ground, sweeping, going downstairs, etc...., in this way we try to simulate as many daily life scenarios as possible. • In addition, the lighting is highly variable, as well as shadows and reflections that can be detected as moving objects.
The videos show people always dressed differently and in four main environments, which are as follows: • Home; • Office; • Reading room; • Coffee room.
This way, the system's independence from the environment can be assessed. Of this dataset, however, not all the videos were used for the realization of the system since they are no longer available; in fact, a total of 201 videos were used; of these, 110 contain falls, and 91 do not contain falls.
This subchapter reviews all stages of creating the system using both 2D and 3D coordinates of the subjects' poses, starting with creating the dataset and arriving at the classification stage. The proposed system scheme is shown in Figure 5.
The videos from the first and second datasets previously described were processed in the following way. The following pre-processing steps were performed for each of these videos. 1. Extraction and storage of 2D/3D coordinates from the video frames; 2. Noise Elimination; 3. Extraction of Spatio-temporal features and sigma log normal; 4. Creation of the final dataset in CSV format.
Each video was divided into frames for the creation and storage of 2D and 3D pose coordinates, and for each frame, the pose of the person present in the frame was estimated. For the extraction of 2D coordinates of the person's body, the OpenPose library was used for the extraction of 3D coordinates, FrankMocap was used. OpenPose represents the first real-time multi-person system to jointly detect key points of the human body, hand, face, and foot on single images; then, given a video, OpenPose divides it into frames and for each one extracts the 2D coordinates of each body part and stores them in a file. In contrast, FrankMocap is a 3D motion capture system that, given a video, transforms it into frames and, for each frame, provides an estimate of 3D poses for a person's body and hands. This system, therefore, makes it possible to obtain estimates of the 3D poses of people within the various frames without having a complex system of cameras capturing the subject from different perspectives.
OpenPose, as we have already mentioned, produces as output a file containing the extracted 2D coordinates for each body part, i.e., the key points. The key points extracted for the body parts of a person are 25 in total. However, not all key points were used to realize the fall detection system. Only 14 key points were used, which are the following: Nose, Neck, right shoulder, RightElbow, RigthWrist, LeftShoulder, LeftElbow, LeftWrist, RightHip, RightKnee RightAnckle, LeftHip, LeftKnee, LeftAnckle. All key points were not used because they were not relevant to the Fall Detection system realizations; in fact, the points that were not taken into account are related to, for example, the ears, heel, eyes, and toes, etc...
In the case where OpenPose could not extract the 2D coordinates of the various key points, since it extracts the coordinates only if they are visible in the frame, these coordinates were interpolated according to the value that the coordinate has in the other frames, in some cases (precisely in 15 videos) instead the videos were too dark and Open Pose was not able to extract any points. For 3D coordinate extraction, on the other hand, as has already been mentioned, FrankMocape was used, and the same key points used for 2D plus MidHip were extracted and used. With FrankMocap, you do not have the problem of null coordinates of key points since it already estimates all coordinates.
For noise elimination in the extracted 2D and 3D key points, the Kalman Filter was applied. The Kalman filter is an algorithm that, given observed measurements over time that contain noise, provides an estimate of values that tend to be more accurate than the actual measurements.
After applying the Kalman filter, 2D and 3D features were computed on the key points of the various body parts. For both 2D and 3D, two types of features were computed, which are as follows: • Spatiotemporal features; • The sigma log-normal features.
Spatiotemporal features make it possible to analyze the movement of people in space and understand when a person is falling. The features calculated for 2D were as follows: Joint Displacement, Displacement in x, Displacement in y, Joint Velocity, Velocity in x, Velocity in y, Joint Acceleration, Acceleration in x, Acceleration in y, and Tangent Angle to Trajectory. For the displacement calculation for each video, the coordinates of the 2D key points in pairs of frames were taken. In contrast, the difference between the time of the first frame and the time of the second frame was taken as the reference time to calculate the various velocities and accelerations. For the 3D key points, on the other hand, the spatiotemporal features calculated were as follows: Joint Displacement, Displacement in x, Displacement in y, Displacement in z, Joint Velocity, Velocity in x, Velocity in y, Velocity in z, Joint Acceleration, Acceleration in x, Acceleration in y, Acceleration in z, Angle tangent to the trajectory. The calculation of these features was performed in the same way as the 2D features, however, using the coordinates of the 3D key points. Normal sigma log features are usually studied in the field of digital signatures: in fact, they are an essential tool of signature recognition as they allow us to learn of the variations of Acceleration and pressure during motion. These features have been used because they allow for the analysis of variations in Acceleration, a fundamental aspect of the knowledge of falls. These types of features are computed for both 3D and 2D for each body part within each frame of the video and associated with spatiotemporal features based on time, while the number of normal sigma logs extracted for each frame of the video is repeated for all body parts in the frame.
After extracting the 2D features for each frame of the videos in the two datasets, they were labeled with one of the videos falling otherwise with 0 and were stored in a CSV file, one for each video. Then two CSV files were created representing the two final datasets for 2D; the first CSV file is the union of the previously created CSV files related to the videos belonging to the first dataset; also, the second CSV file is the union of all the CSV files created previously however associated with the videos belonging to the second dataset. For the creation of the final datasets with 3D coordinates, the same procedure described above was carried out as for the 2D coordinates.
A Neural Network with the following architecture was used as the classifier. Its architecture is shown in Fig. 6. The 2D and 3D datasets were given as input to the neural network in the form of a vector of sequences, where each sequence is a video, i.e., the set of appropriately labeled frames that make up the video itself. Padding was performed to ensure that all the sequences had the same size by adding zeros to the beginning of the sequences. As can be seen from the figure above showing the network architecture used, we have a masking layer, which is used to disregard the various zeros inserted in the sequences to make sure that the various sequences have the same length; then we have an LSTM layer with some neurons equal to 64, a Dropout layer to reduce overfitting of the model with a rate of 0. 2, the Attention layer with 128 neurons, a dense layer with 34 neurons a Normal initializer kernel and the L2 regularize that further decreases the overfitting, finally we have a Dens layer with two neurons, as we have two classes dropped and not dropped, with SoftMax activation function.
This chapter presents the conducted experiments and the results obtained using training validation, testing strategies, and the databases created from 2D and 3D coordinates. The experiments evaluate the system performance with and without Attention Mechanism in term of Precision, Sensitivity, Specificity, F1-measure, Accuracy and AUC_ROC.
The two previously created datasets for both 2D and 3D were used for the Training Validation and Test phase. Two strategies were used for this phase:
The first strategy is to merge the two datasets into one dataset and then divide them into three parts for each phase (Training, Validation, and Test). The split made was as follows (75% of the initial dataset for the Training phase, 12% of the initial dataset for the Validation phase, and 13% of the initial dataset for the Test phase). On the other hand, the second strategy uses the first dataset for the Training and Validation phase, precisely 77% for Training and 23% for Validation, and the second dataset for the Testing phase.
With both the first and second strategies during the training phase, the model was trained on 150 epochs on the training set; at each epoch, Validation was carried out with the validation set and stored the model with the highest accuracy among the epochs. At the end of Training and Validation, testing was carried out with the test set, using the best model obtained among the epochs. Two strategies were used for this phase, performing both an intra-dataset and interdataset test to see the model's behavior on completely different videos than those used for the training and validation phase. For the testing phase, the metrics used were as follows: Precision, Accuracy, Sensitivity, Specificity, F1-measure, and AUC_ROC.
The first strategy, described in the previous chapter, of combining the two datasets into a single dataset and then dividing it into three parts one for each phase (Training, Validation, and Testing) was repeated 20 times first using the neural network with Attention and then using the neural network without Attention. The average results obtained over the 20 repetitions with the dataset created from the 2D coordinates are as follows (Table 2).
The results obtained with the dataset created from the 3D coordinates are as follows (Table Without Attention 0.772 +/- 0.12 0.807 +/- 0.109 0.73 +/- 0.167 0.778 +/- 0.065 0.768 +/- 0.077 0.768 +/- 0.077
The results obtained using the second strategy for the training validation and testing phase, the one described in the previous paragraph, which is to use the two initial datasets separately one for the training and validation phase and the other for the testing phase, were as follows: Using the network with Attention and the network without attention as a classifier with the 2D dataset were as follows (Table 4).
In contrast, the results obtained with the 3D dataset are as follows (Table 5).
Specificity F1-measure Accuracy AUC_ROC
As can be seen, in the second strategy it is impossible to have average results because a separate dataset was used as the test set, consequently, the examples in the test set would not change if more iterations were performed. We summarize the results obtained in the following graphs to get a more immediate view of the results obtained, as shown in Figures 7, 8, 9 and 10.
The neural network with attention performs significantly better than the neural network without attention, using both the first and second strategies for training validation and testing, as shown in Figures 7, 8, 9 and 10.
Table 6 and Table 7 present state-of-the-art comparisons of the same datasets. Unfortunately, it is not easy to compare accuracies because of the different settings of the tests: some works used 80% of the dataset instances for training and 20% for testing, while in our work we used a 10Fold cross-validation setting. Looking at the results on the URFD dataset it is possible to observe that our solution achieved comparable results concerning other solutions. On the Le2I dataset, instead, our solution both in 2D and 3D achieved inferior results. This can be mainly imputed to the decided experimental setting which was selected to decrease the chance of bias and create inaccurate predictions.
In this paper two Fall Detection systems have been proposed. The first system uses 2D poseestimating subjects, and the second uses 3D pose-estimating subjects by a neural network with attention and without attention. The neural network with attention achieves significantly better results than the neural network without attention, using both the first and second strategies for training validation and testing (Fig. 6-7). Note especially that in the second strategy, where an entire dataset is used as a test set, there are also good results here with both 2D and 3D. In fact, with the 2D dataset, have an Accuracy and Precision, using the neural network with attention of 72% and 63% respectively, while not using attention, an accuracy of 52% and a Precision of 0% have been obtained. In contrast, with the 3D dataset, using the neural network with attention, we achieve an Accuracy of 81% and a Precision of 90%. In comparison, without attention, we achieve an Accuracy of 70% and a Precision of 62%. This shows us that the model with attention can be generalized because it does not bind to Training data. In addition, the model created with the attention mechanism is also able to generalize because it does not suffer from scene bias since, although there are many different scenarios in the videos, the attention mechanism manages to put "attention" on the subject present in the video while neglecting the context/scenario. It is also possible to observe that applying the Attention mechanism to perform time series modeling of body joint coordinates results in a sensible jump in accuracies, showing that temporal pattern weighting is a winning strategy when it comes to fall detection and, in general, activity recognition. In the future, other state-of-the-art technologies such as the Multi-Speed Transformer, which uses the multiresolution intuition to model time series classification with a fast and a slow branch, will be used in the same conditions to improve the fall detection performance.
Francesco Castro is a PhD student enrolled in the National PhD in Artificial Intelligence, XXXVIII cycle, course on Health and life sciences, organized by Università Campus Bio-Medico di Roma. The research of Dott. Vincenzo Gattulli was funded by PON Ricerca e Innovazione 2014-2020 FSE REACT-EU, Azione IV.4 "Dottorati e contratti di ricerca su tematiche dell'innovazione" CUP H99J21010060001.
This project is supported by the AMICA project.