This research paper introduces a novel approach to address privacy concerns in video-based elderly fall detection systems without compromising such technologies' efficacy and real-time response. The methodology integrates EfficientNetB0 for robust feature extraction from video sequences and Long Short-Term Memory networks for accurate fall classification. Despite achieving exemplary performance metrics, including 100% scores in accuracy, Area Under the Curve, recall, and Precision, the pervasive issue of privacy infringement in video surveillance remains a significant challenge. To tackle this, we propose a dynamic blurring technique that selectively obscures identifiable features within video frames, such as faces and distinguishing clothing, thus maintaining individual anonymity. This method ensures that the privacy of the monitored individuals is preserved while retaining the essential details necessary for the fall detection algorithm to function effectively. This paper details this privacypreserving technique and demonstrates its feasibility without detracting from the system's performance. Our findings indicate that integrating dynamic blurring into the fall detection pipeline offers a promising solution to the privacy concerns associated with video-based monitoring systems. It protects sensitive personal information while providing high care and safety. This research contributes to the broader discourse on ethical technology use in healthcare. Moreover, it emphasizes the importance of balancing advanced monitoring capabilities with the imperative of privacy preservation.
The growing demographic of the elderly population has precipitated an increased incidence of
falls [
There has been a significant relationship between the efficacy of video-based fall detection and
the imperative to protect individual privacy [
In response to these challenges, this paper proposes an innovative solution that retains the advantages of video surveillance while addressing privacy concerns. Our approach employs dynamic blurring, selectively obscuring identifiable features within video frames. Thus, individuals are anonymized without compromising the system’s ability to detect falls. This 0009-0002-9879-8008 (I. Ursul); 0009-0007-1598-8161 (J. H. Muzamal); © 2024 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
method differs from existing solutions by offering a real-time, privacy-preserving mechanism
that does not detract from the system’s performance. Integrating EfficientNetB0 [
This research aims to develop a fall detection system that fulfills the need for efficient, realtime monitoring with the imperative of privacy preservation. Our objectives include designing and implementing a dynamic blurring technique within a video-based fall detection framework. Moreover, we also aim to evaluate this system’s accuracy and privacy protection performance and demonstrate its applicability in real-world settings. This research can potentially contribute to the development of ethically responsible technological solutions in healthcare, particularly in the context of elderly care. This work seeks to pave the way for broader acceptance by addressing the privacy concerns associated with video-based monitoring. Moreover, deploying such systems enhances the safety and well-being of the elderly population.
The exploration of fall detection systems, particularly for the elderly, is an area of research that has seen substantial evolution over time. Dean et al. [19] 2006 implemented the first real-time fall detection system using a triaxial accelerometer. At that time, most traditional techniques centered around simplistic, mechanical solutions and gradually transitioned towards incorporating technology [20]. Among the earliest methods were basic alert systems, which relied on the user to trigger an alert manually in case of a fall [21]. While pioneering for their time, these systems were limited by their dependence on the users to activate the alarm post-fall, which could be compromised due to injury.
Advancements in technology brought in a new wave of methodologies, primarily categorized
into sensor-based [
Traditional algorithms such as Support Vector Machines (SVMs) [30] and Decision Trees [31] were widely employed in the early stages of machine learning applications for fall detection. These methods primarily relied on handcrafted features extracted from sensor data or basic video analytics, including motion vectors and silhouette shapes. Flow-based methods, particularly optical flow [32], were also prominent, enabling the detection of movement patterns by analyzing the apparent motion of objects, surfaces, and edges. While effective to a certain extent, these approaches faced limitations in handling the high variability and complexity of human falls. They often struggled to distinguish falls from other activities involving rapid movements, leading to high false alarm rates [33], [34]. Additionally, their dependency on manually crafted features restricted their adaptability, as these features might not generalize well across different scenarios.
The recent emergence of deep learning architectures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) has changed many dynamics [35]. Advanced models such as ResNet [36], LSTM [37], and YOLO especially marked a leap forward in fall detection. CNNs, with their ability to perform automatic feature extraction, have proven particularly adept at analyzing spatial characteristics in video frames [38]. While RNNs and LSTMs excel in capturing temporal dependencies, it is crucial to understand the sequence of movements leading to a fall. YOLO [39], an object detection model, brought further advancements by enabling real-time processing. Despite their successes, the search for enhanced performance led to exploring hybrid methods that combine multiple deep learning models. For instance, integrating CNNs with LSTMs allows for the effective processing of video data both spatially and temporally, offering a better understanding of fall events [40]. These hybrid approaches [41], alongside innovative methods within deep learning frameworks, promise to address the dynamics of fall detection [42].
Recent advancements aim to address these privacy concerns while maintaining system efficacy. Techniques such as dynamic blurring and real-time anonymization have been explored to obscure identifiable features in video feeds. This can help safeguard individual privacy without significantly compromising detection capabilities. Despite these efforts, there is a gap in the literature concerning developing a system that seamlessly integrates high detection accuracy with robust privacy protection. Our contribution to this field addresses this gap by proposing a novel fall detection system that employs EfficientNetB0 for advanced feature extraction and LSTM networks for accurate temporal classification, complemented by a dynamic blurring mechanism to ensure privacy. This integrated approach promises high performance, as evidenced by optimal accuracy, recall, and precision scores. Moreover, it introduces a viable solution to the privacy concerns that have long shadowed video-based monitoring systems. By achieving this delicate balance, our research paves the way for the broader acceptance of videobased fall detection systems, ensuring the safety of the elderly population.
This research presents a methodological framework to address the challenge of detecting falls through video surveillance while safeguarding their privacy. The foundation of the proposed approach is mathematical models and techniques to ensure precision, efficiency, and reliability. The proposed method integrates state-of-the-art EfficientNetB0 for spatial feature extraction and LSTM networks for temporal sequence analysis. Additionally, we introduce a dynamic blurring mechanism formulated to preserve privacy by selectively obscuring identifiable features within video frames. Figure 1. provides the overall architecture of the proposed approach.
The dataset employed in this study was sourced from the UR Fall Detection Dataset [43], encompassing 70 sequences, of which 30 are fall events, and 40 represent activities of daily living (ADL). The fall events were captured using two Microsoft Kinect cameras, accompanied by accelerometric data, whereas the ADL events were documented using a single camera (camera 0) alongside accelerometer data. The accelerometric data was acquired through PS Move (60Hz) and x-IMU (256Hz) devices. The dataset is structured such that each sequence comprises depth and RGB images from both camera perspectives (parallel to the floor and ceiling-mounted), synchronization data, and raw accelerometer readings. Each video stream is archived separately as a sequence of PNG images. The depth data, stored in PNG16 format, necessitates rescaling to represent depth in millimeters (D) as follows accurately: ( , ) = ( , )⋅ 65535 (1)
Where ( , ) denotes the depth at position ( , ) for the ith camera, ( , ) represents the pixel value at position ( , ) in the PNG16 image, and is the scale ratio for the i-th camera. The scale ratios are defined as 0 = 6000 for fall sequences using camera 0, 1 = 3640 for fall sequences using camera 1, and 0 = 7000 for ADL sequences using camera 0. The preprocessing of video data involves a series of steps to prepare the frames for feature extraction. Initially, each video is accessed frame by frame using OpenCV’s VideoCapture functionality. Subsequently, each frame is resized to a uniform dimension of 224 × 224 pixels to align with the input requirements of the EfficientNetB0 model. This resizing operation can be mathematically represented as a function R that maps the original frame dimensions to the target dimensions, preserving the aspect ratio and interpolating pixel values as necessary:
Where w and ℎ denote the original width and height of the frame, respectively. After resizing,
the frames undergo normalization to scale the pixel values to the [
225
This operation ensures that each pixel value in the frame is proportionally reduced to a decimal between 0 and 1, thus standardizing the input data for subsequent processing through the EfficientNetB0 architecture.
The feature extraction component of our methodology is built upon the EfficientNetB0 architecture, a cutting-edge CNN known for its scalability and efficiency. EfficientNetB0 uniformly scales the network’s depth, width, and resolution, optimizing its performance across various constraints. EfficientNetB0 are its convolutional operations, which form the backbone of its feature extraction capabilities. A convolutional operation on an input image or feature map can be mathematically described as: (2) (3) (4) (5) ( , ) = ∑ = − ∑ = − ( , ) ⋅ ( − , − )
Where Fout is the output feature map, Fin is the input image or feature map, K is the kernel or filter of size (2 + 1) × (2 + 1), and ( , ) denotes the pixel coordinates. This operation is applied across the entire input feature map, extracting features through the weighted summation of pixel values
within the kernel’s receptive field. EfficientNetB0 also leverages batch normalization to enhance training stability and convergence. Batch normalization can be defined as: ( ) = ( −
) + √ 2 +
Where x is the input to the batch normalization layer, μB and σB2 are the mean and variance of the batch, respectively, γ and β are learnable parameters of the layer, and ϵ is a small constant added for numerical stability. Furthermore, EfficientNetB0 employs depthwise separable convolutions, a technique that reduces computational cost without sacrificing depth or expressivity. A depthwise separable convolution comprises two stages: depthwise and pointwise convolution. The depthwise convolution applies a single filter per input channel, and the pointwise convolution then combines the output channels using a 1×11×1 convolution. This can be represented as: (6) (7)
Where DW denotes the output of the depthwise convolution for channel c, PW is the output of the pointwise convolution for channel c′, is the kernel for the depthwise convolution, and K′ is the 1×11×1 kernel for the pointwise convolution. C is the number of channels. Activation functions such as the Swish function, defined as ( ) = ⋅ ( ), are applied after convolutional operations to introduce non-linearity, enabling the network to learn complex features. By integrating these elements, EfficientNetB0 provides a powerful and efficient framework.
3.3. Dynamic Blurring for Privacy Preservation we address privacy concerns in video-based monitoring by implementing dynamic blurring for privacy preservation. This process involves the selective obfuscation of regions of interest (ROI) within video frames, specifically targeting identifiable features of individuals to maintain anonymity while preserving the utility of the data for fall detection. The identification of ROIs for blurring is governed by a detection function ( , ), where represents an input frame, and denotes the parameters of the detection model, which may include facial recognition, pose estimation, or other relevant feature detection algorithms. The output of this function is a set of bounding boxes
= { 1, 2, . . . , }, where each specifies the coordinates and dimensions of an ROI within the frame. The dynamic blurring is then applied to these identified ROIs using a Gaussian blur operation, mathematically described as: ( , , ) = 2 1 2 − 2 2 2+ 2 (8) within the frame can be represented as:
Where ( , ) are the coordinates relative to the center of the kernel, and is the standard deviation, which controls the extent of blurring. The size of the kernel, × , is chosen based on the desired level of blurriness, typically set to several times the value of to ensure that the edges of the kernel contribute negligibly to the blur. The application of the Gaussian blur to an ROI ( , ) = ( ∗ )( , ) = ∑
= − ∑ = − ( − , − ) ⋅ ( , , ) (9) width and height of the Gaussian kernel, respectively.
for all ( , ) within , where ∗ denotes the convolution operation, and a and b are half the
The LSTM network is a specialized RNN designed to model temporal dependencies in sequence data effectively. Its architecture is uniquely suited to address the vanishing gradient problems, enabling it to capture long-term dependencies. An LSTM unit comprises three main gates: the input gate (i), the forget gate (f), and the output gate (o), each responsible for regulating the flow of information. We utilized bidirectional LSTM with the following structure, as shown in Figure
Following the extraction of temporal features, the next step involves classification, which is classified between fall and non-fall events. This process typically involves passing the LSTM output through one or more fully connected layers in a SoftMax layer for binary classification: = ℎ = ℎ ⋅ ℎ + ℎ ( ) = ∑ (10) (11) where ℎ is the output from the LSTM at time t, ℎ and ℎ are the weights and biases for the dense layer, respectively, z is the logit, and p represents the probabilities for each class obtained through the SoftMax function. The class with the highest probability is selected as the predicted class for each input sequence. This framework facilitates the effective classification of video sequences into fall or non-fall categories based on the temporal patterns identified by the LSTM network.
The training of the integrated model is underpinned by a mathematical framework that includes the definition of a loss function, the selection of an optimization algorithm, and the application of regularization techniques to prevent overfitting. The loss function L quantifies the discrepancy between the predicted outputs p and the true labels y. For binary classification tasks, such as fall detection, the binary cross-entropy loss is commonly used: ( , ) 1 ∑ =1 [ log( ) + (1 − ) log(1 − )] (15)
Where N is the number of samples, is the true label, and is the predicted probability for the i-th sample. The optimization of the model parameters is achieved through Stochastic Gradient Descent (SGD), which iteratively updates the weights W based on the gradients of the loss function: +1 = − (16)
Where is the learning rate, and ∇L denotes the gradient of the loss function for the weights at time t. L2 regularization and dropout techniques are applied to mitigate overfitting by adding a penalty term to the loss function or randomly omitting units from the network during training, respectively. The backpropagation process facilitates the computation of gradients ∇L through the network, employing the chain rule to propagate errors from the output layer back through the LSTM and EfficientNetB0 layers, enabling the model to learn and adjust its parameters to minimize the loss function.
The confusion matrix in Figure 4. provides a quantitative assessment of the model’s classification accuracy. It shows the number of true positive (TP) and true negative (TN) predictions, along with false positive (FP) and false negative (FN) predictions. In this case, the matrix reveals perfect classification on the test data, with all fall events being correctly identified (6 TP) and no ADL events being misclassified as falls (0 FN), implying an exceptional level of model performance.
The Receiver Operating Characteristic (ROC) curve in Figure 5. plots the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. The area under the curve (AUC) in this ROC curve approaches 1, which suggests excellent model performance, with a high true positive rate and a low false positive rate across threshold values.
Figure 6 shows the model’s performance on real test videos; the model correctly predicted this particular scenario as a ‘Fall,’ corroborated by the RGB image on the right. This clearly shows an individual in a prone position on the floor. Moreover, the depth image reveals the successful application of the dynamic blurring method. The individual’s features are indistinguishable, and the privacy-preserving objective of the method is evident. The contours and the general posture of the person are discernible, which is sufficient for fall detection purposes, but the finer details necessary for personal identification have been effectively obfuscated. The blurring technique implemented in the system is designed to activate upon detecting a human figure within the video frame, applying a Gaussian blur where the person is detected. This ensures that any potentially sensitive information is rendered non-identifiable, addressing privacy concerns paramount in real-world applications of surveillance-based systems. The obscured depth image confirms that the privacy-preserving measures do not impede the algorithm’s ability to detect a fall.
Figure 7 shows the system’s prediction for this scene, labeled ‘ADL,’ which is validated by the RGB image on the right. It depicts an individual in an upright position, supporting the prediction that no fall has occurred. The prediction’s accuracy is a testament to the model’s ability to effectively discern between falls and non-fall events. Furthermore, similar to the previous fall scenario, the depth image demonstrates the application of the dynamic blurring technique. The individual’s detailed features are indistinct, ensuring privacy is maintained. Despite the blurring, essential characteristics for ADL recognition, such as the vertical orientation of the body and the absence of unusual postures associated with falls, are preserved and remain detectable by the system.
The analysis of the presented results underscores the robustness and reliability of the implemented fall detection model. This is evidenced by the convergence of the accuracy and loss metrics, the unequivocal classification outcomes depicted in the confusion matrix, and the favorable diagnostic characteristics portrayed by the ROC curve. These results collectively affirm the model’s efficacy in accurately detecting fall events. It preserves the privacy of individuals through dynamic blurring, as no identifiable features are discernible in the depth visualizations.
As shown in Table 1, the comparative analysis of fall detection methodologies yields a substantive understanding of the advancements and varying efficacies of diverse approaches in this research domain. The table encapsulates the True Positive Rate (TPR), True Negative Rate (TNR), and overall Accuracy, serving as pivotal metrics for the assessment of each method. The model presented by Eltahir et al. [40] manifests a commendable balance between sensitivity and specificity, with a TPR of 95.88% and a TNR of 97.02%, culminating in an accuracy of 97.56%. Chan Su’s model slightly improves sensitivity at 98.07% and specificity at 99.03%, with an analogous accuracy of 98.06%. These two models set a robust baseline in fall detection, evidencing high efficacy. The single-stream models using RGB and Optical Flow (OF) data individually attain a TPR of 100%, indicative of their flawless identification of fall events. However, their specificity scores, 96.61% and 96.34%, respectively, although high, suggest a slightly less robust capacity to classify non-fall activities accurately. This slight discrepancy is reflected in their accuracy scores, which, while impressive at 96.99% and 96.75%, do not reach the pinnacle of Chan Su’s model.
The multi-stream approach amalgamating RGB, OF, and Pose Estimation (PE) data represents a significant leap forward, yielding a perfect TPR and an enhanced TNR of 98.61%, leading to an accuracy of 98.77%. This approach underscores the utility of integrating multiple data streams for improved specificity without compromising sensitivity. EfficientNet-B0, despite a lower TPR of 93.33%, achieves a perfect TNR of 100%. This accentuates the model’s exceptional performance in identifying non-fall events, though it falls short of the multi-stream model’s balanced accuracy. The improved YOLOv5s model and the single frame human binary image approach using YOLOv5s do not disclose TPR or TNR but report accuracies of 97.2% and 96.7%, respectively. While these figures suggest competent models, the lack of detailed TPR and TNR data precludes a complete comparative analysis. Our proposed methodology establishes a new benchmark, recording a flawless TPR and TNR of 100% and an unmatched accuracy of 100%. This unprecedented performance indicates a superior ability to correctly identify fall incidents and an unparalleled precision in confirming non-fall activities.
In this research, we have successfully developed and evaluated a novel video-based fall detection system that prioritizes privacy without compromising the real-time detection efficacy of elderly falls. By integrating EfficientNetB0 and LSTM networks, our methodology ensures robust feature extraction and accurate fall event classification. The introduction of dynamic blurring as a privacy-preserving technique represents a significant advancement, allowing for anonymizing identifiable features within video frames while maintaining the system’s operational integrity. Our findings reveal that this approach achieves perfect accuracy, recall, precision, and area AUC scores. It also effectively addresses the critical privacy concerns of video surveillance in sensitive environments such as homes and elderly care facilities. Implementing dynamic blurring ensures that the privacy of monitored individuals is safeguarded, setting a new precedent in the ethical application of surveillance technologies in healthcare.
Our future research will focus on further enhancing the adaptability and generalizability of the system across diverse settings and populations. This includes exploring additional privacypreserving mechanisms and integrating multimodal data sources to enrich the system’s contextual understanding. This research contributes significantly to elderly care technology, presenting a practical solution to the long-standing challenge of balancing effective fall detection with stringent privacy requirements. Our work advances the technological capabilities in this domain and addresses critical ethical considerations. This paves the way for broader acceptance and deployment of video-based monitoring systems in healthcare settings.
S. A. Carneiro, G. P. da Silva, G. V. Leite, R. Moreno, S. J. F. Guimaraes, and H. Pedrini, “Multistream deep convolutional network using high-level features applied to fall detection in video sequences,” in 2019 International Conference on Systems, Signals and Image Processing (IWSSIP), IEEE, 2019, pp. 293–298. Accessed: Mar. 18, 2024.
N. Kaur, S. Rani, and S. Kaur, “Real-time video surveillance based human fall detection system using hybrid haar cascade classifier,” Multimed. Tools Appl., pp. 1–19, 2024.
A. Núñez-Marcos and I. Arganda-Carreras, “Transformer-based fall detection in videos,” Eng. Appl. Artif. Intell., vol. 132, p. 107937, 2024.
J. Moore et al., “Contextualizing remote fall risk: Video data capture and implementing ethical AI,” NPJ Digit. Med., vol. 7, no. 1, p. 61, 2024.
V. Fula and P. Moreno, “Wrist-Based Fall Detection: Towards Generalization across Datasets,” Sensors, vol. 24, no. 5, p. 1679, 2024.
A. Bansal, R. Sharma, and M. Kathuria, “A Vision-Based Approach to Enhance Fall Detection with Fine-Tuned Faster R-CNN,” in 2023 International Conference on Advanced Computing & Communication Technologies (ICACCTech), IEEE, 2023, pp. 678–684.
J. Gutiérrez, V. Rodríguez, and S. Martin, “Comprehensive review of vision-based fall detection systems,” Sensors, vol. 21, no. 3, p. 947, 2021.
S. Ezatzadeh and M. R. Keyvanpour, “Fall detection for elderly in assisted environments: Video surveillance systems and challenges,” in 2017 9th international conference on information and knowledge technology (ikt), IEEE, 2017, pp. 93–98. Accessed: Mar. 19, 2024.
I. Charfi, J. Miteran, J. Dubois, M. Atri, and R. Tourki, “Optimized spatio-temporal descriptors for real-time fall detection: comparison of support vector machine and Adaboostbased classification,” J. Electron. Imaging, vol. 22, no. 4, pp. 041106–041106, 2013.
F.-Y. Leu, C.-Y. Ko, Y.-C. Lin, H. Susanto, and H.-C. Yu, “Fall detection and motion classification by using decision tree on mobile phone,” in Smart Sensors Networks, Elsevier, 2017, pp. 205–237. Accessed: Mar. 19, 2024.
Y.-Z. Hsieh and Y.-L. Jeng, “Development of home intelligent fall detection IoT system based on feedback optical flow convolutional neural network,” Ieee Access, vol. 6, pp. 6048– 6057, 2017.
P. Vallabh and R. Malekian, “Fall detection monitoring systems: a comprehensive review,” J. Ambient Intell. Humaniz. Comput., vol. 9, no. 6, pp. 1809–1833, 2018.
R. Igual, C. Medrano, and I. Plaza, “Challenges, issues and trends in fall detection systems,” Biomed. Eng. OnLine, vol. 12, no. 1, p. 66, 2013, doi: 10.1186/1475-925X-12-66.
Y. Fan, M. D. Levine, G. Wen, and S. Qiu, “A deep neural network for real-time detection of falling humans in naturally occurring scenes,” Neurocomputing, vol. 260, pp. 43–58, 2017.
D. Singh, M. Gupta, and R. Kumar, “BGR Images-Based Human Fall Detection Using ResNet50 and LSTM,” in Third Congress on Intelligent Systems, vol. 608, S. Kumar, H. Sharma, K. Balachandran, J. H. Kim, and J. C. Bansal, Eds., in Lecture Notes in Networks and Systems, vol. 608. , Singapore: Springer Nature Singapore, 2023, pp. 175–186. doi: 10.1007/978-981-199225-4_14.
N. Lu, Y. Wu, L. Feng, and J. Song, “Deep learning for fall detection: Three-dimensional CNN combined with LSTM on video kinematic data,” IEEE J. Biomed. Health Inform., vol. 23, no. 1, pp. 314–323, 2018.
C. Su, J. Wei, D. Lin, L. Kong, and Y. L. Guan, “A novel model for fall detection and action recognition combined lightweight 3D-CNN and convolutional LSTM networks,” Pattern Anal. Appl., vol. 27, no. 1, pp. 1–16, 2024.
T. Chen, Z. Ding, and B. Li, “Elderly fall detection based on improved YOLOv5s network,” IEEE Access, vol. 10, pp. 91273–91282, 2022.
M. M. Eltahir et al., “Deep Transfer Learning-Enabled Activity Identification and Fall Detection for Disabled People.,” Comput. Mater. Contin., vol. 75, no. 2, 2023, Accessed: Mar. 18, 2024.
S. Hwang, M. Ki, S.-H. Lee, S. Park, and B.-K. Jeon, “Cut and continuous paste towards realtime deep fall detection,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. 1775–1779. Accessed: Mar. 18, 2024.
Y. Wang, R. Song, and X. Zhang, “Real-time human fall recognition based on deep learning methods and single depth image with privacy requirements,” in 2022 37th Youth Academic [37] [38] [39] [40] [41] [42]