Blood pressure (BP) estimation is a standard and critical component of routine health assessment, especially for cardiac disease patients. Traditional methods typically require direct contact with the patient, which can cause discomfort and inconvenience. Remote photoplethysmography (rPPG) that enables non-contact measurement of the blood volume pulse using trivial cues from facial videos has drawn attention to measure vital signs. This paper presents an ensemble deep learning approach for estimating BP remotely using facial videos. Specifically, to address the vulnerabilities and biases in deep learning models for BP measurement, we emphasize both the accuracy of individual models and the diversity within the ensemble. We utilize advanced deep learning architectures to construct several regression models incorporating convolutional neural networks and transformer blocks, which learn the spatiotemporal relationships between diferent frames and locations. These trained models are then combined to measure BP readings. Additionally, to enhance the system's robustness under varying lighting conditions, data augmentation techniques are employed to generate more training data. The proposed method is tested on an unseen dataset and the average root of mean squared error (RMSE) is 12.95 mmHg, ranking 1st in the 3rd Vision-based Remote Physiological Signal Sensing (RePSS) Challenge.
CEUR ceur-ws.org
Blood pressure (BP) measurement is a fundamental diagnostic tool in medical practice, serving
as a crucial indicator of cardiovascular health. For instance, elevated BP, or hypertension, is
a significant risk factor for cardiovascular diseases, including stroke, heart attack, and renal
failure, making accurate and timely measurement vital for early detection and management [
Currently, cufless BP monitor methods have been explored for real-time measurement, and
providing convenience and comfort. There are two main methods: pulse transit time (PTT) and
pulse wave analysis (PWA) [
Note that these methods are contact-based and require specific devices like smart watches
to make the measurement. Over the past years, remote PPG (rPPG) techniques have been
developed for vital sign measurements, especially for heart rate (HR) estimation [
This paper proposes to achieve rPPG-based BP measurement with ensemble deep learning using facial videos. Specifically, to address the vulnerabilities and biases in deep learning models for BP measurement, we prioritize the accuracy of individual models and the diversity within the ensemble. We construct individual regression models by adding a regression head to CNNand transformer-based backbones. For training each model, we use not only the original RGB images but also features obtained by transforming the color space from RGB to YUV. To enhance the models’ performance under varying lighting conditions, data augmentation techniques are employed. Finally, an aggregator is used to combine the outputs from these individual models.
Invasive BP estimation can provide continuous and accurate monitoring and is there essential in
certain clinical settings, particularly for patients under critical care or during surgery [
Cuf-based BP measurement is the most common non-invasive method used in both clinical
and home settings to assess ABP [
PPG-based BP estimation is getting more widely used as the emergence of deep learning
algorithms and PPG sensors that can be placed on the finger, earlobe, or over the wrist [ 18].
Variations in light absorption during the cardiac cycle are measured, providing information about
the blood flow, heart rate, and other cardiovascular attributes. By analyzing these variations,
algorithms can estimate systolic and diastolic BP values [
Recently, the rPPG-based methods ofer a non-contact way for BP estimation by using video cameras to detect blood volume changes in facial skin [20]. This technology, which can be implemented with standard RGB cameras found in common devices like smartphones and tablets, captures subtle changes in light reflection of the skin due to pulsating blood flows [ 21, 22, 23]. The rPPG-based methods are non-invasive and use widely-accessible cameras, making it potentially cost-efective and convenient for regular BP checks. However, the accuracy can be compromised by factors such as motion and variable lighting conditions, posing challenges for its use in dynamic or uncontrolled environments.
The overall framework of our ensemble deep learning method is illustrated in Fig. 1, from which we can see that there are multiple regression models. To import diversity, multiple models are trained using diferent input feature vectors, backbones, or random seeds. The outputs of individual models are then fused with an aggregator.
A short clip is extracted from the original full video and then partitioned into frames. It is worth pointing out that we select the clip closest to the time when BP is measured to mitigate the impact of BP fluctuation during video taking. If the video is recorded before BP measurement, the last part the video is selected and vice versa for videos taken after BP measurement. The face region of each frame is then cropped and resized to 128 × 128. To improve model performance in diferent lighting conditions, data augmentation technique is applied during the training process.
It has been demonstrated in [
YUV conversion
Head Regressor 2, seed 1
PhysFormer … Regressor 4N-1, seed N Regressor 4N, seed N
Aggregator
Aggregated SBP/DBP RGB frames where , , and represent the red, green, and blue color components of an image, respectively. represents the luminance component, while and represent the chrominance components, capturing the color information minus the brightness.
We utilize two state-of-the-art models as the backbone for our BP estimation model, including a
3D CNN model named PhysNet [25] and a transformer-based model named PhysFormer [
We stack a regression head with one hidden layer on top of the backbone and the regression head has two output nodes corresponding to SBP and DBP, respectively. The regression head can be formulated as h = ( W(1)x + b(1)) y = W(2)h + b(2) where is the standard Sigmoid function. W and b are the weights and biases, respectively. x is the output signal from the backbone. h denotes the vector at the hidden layer. y denotes the output vector consisting of DBP and SBP.
The average RMSE of SBP and DBP is used as the loss function to train our models, defined as = 0.5 ×
∑ =1 (
− )2 √ + 0.5 × ∑ =1 ( − 2 ) √ where and are the ground-truth DBP and SBP results of the ℎ sample, respectively. and are the predicted DBP and SBP of the ℎ sample, respectively. is the number of samples.
As mentioned above, multiple individual models are trained with diferent input features (RGB or YUV), backbones (PhysNet or PhysFormer) and random seeds to introduce diversity to our ensemble method. Ensemble learning is used to aggregate the outputs of individual models. For each sample, we remove the top- and bottom- results and then calculate the average of the rest outputs as ens = ens = 1 1 − 2 − 2 − ∑ =+1 − ∑ =+1 where ens and ens are the aggregated prediction of DBP and SBP, respectively. represent the predicted DBP and SBP of the ℎ model when they are arranged in ascending and order. is the number of individual models. The top- and bottom- values are neglected.
The proposed method is tested using PyTorch on a server equipped with Intel(R) Xeon(R) Gold 6430 CPU and RTX4090 GPU. The models are trained for 150 epochs using AdamW optimizer [28] with learning rate = 1 × 10 −5 and ℎ _ = 1 × 10 −5. The value of in Equ. (4) is set as 3. (2) (3) (4)
Two datasets are used for model training and validation, including the VV-medium dataset [29] and our private dataset. A brief summary of these two datasets is reported in Table 1 and distribution are illustrated in Fig. 2. VV-medium dataset [29] has more videos than BP label because each BP label corresponds to multiple videos. It is shown that BP of VV-medium dataset [29] is more diversely distributed compared to our dataset.
For testing, the OBF Database – Oulu BioFace Database [30, 31] consisting of 100 subjects and 200 facial videos with DBP/SBP labels is used for evaluation. Note that for testing, we only have access to the facial videos and have no access to the ground-truth BP labels.
Three metrics, including the root of mean squared error (RMSE), mean absolute error (MAE) and Pearson correlation coeficient , are used to evaluate model performance on the validation dataset, which are defined as MAE = 1 = √ ∑ =1 ( − )2
=1
∑ | − | ∑
=1 ( − )( − ) √∑=1 ( − ) 2 ∑ =1 ( − ) 2 where and are the ground-truth and predicted SBP/DBP, respectively. is the number of samples. and indicate the average values of ground-truth and prediction, respectively.
For testing, the average RMSE of DBP and SBP is used to evaluate model performance , calculated as
RMSEavg = 0.5 × RMSE + 0.5 × RMSE (5) (6) where RMSE and RMSE are the RMSE of DBP and SBP, respectively.
We randomly split the available dataset into training dataset (80%) and validation dataset (20%). The learning curve one of our individual models is shown in Fig. 3 and it shows that the model converges well in 150 epochs. The scatter plots on validation set are illustrated in strongly correlated and the errors of most samples are within ±10 mmHg.
The RMSE of DBP and SBP are 8.93 mmHg and 11.03 mmHg, respectively. The MAE and RMSE of SBP are larger than those of DBP because SBP range is larger and more diversely distributed, which can be seen from Fig. 2. On the testing dataset, the average RMSE is 12.95 mmHg and a comparison is reported in Table 2. One can see that our method outperforms competing methods by more than 0.5 mmHg.
This paper presented an ensemble deep learning method for BP estimation using facial videos. To improve the diversity of models to ensemble learning, multiple models are built with diferent backbones and input feature vectors. Besides, data augmentation technique is used to improve model performance under diferent lighting conditions. The outputs of individual models are fused with an aggregator. Our method is tested on an unseen dataset in the RePSS challenge, and the average RMSE of SBP and DBP is 12.95 mmHg, which outperforms all the peer methods and indicates the efectiveness of our proposed method.
This work is supported by A*STAR Gap project Face AI (Phase 1) under project No. SC36/19000801-A042 and A*STAR Career Development Fund under grant No. C233312006. an observational study, Critical care 18 (2014) 1–11. [16] P. Palatini, R. Asmar, Cuf challenges in blood pressure measurement, The Journal of
Clinical Hypertension 20 (2018) 1100–1103. [17] M. Forouzanfar, H. R. Dajani, V. Z. Groza, M. Bolic, S. Rajan, I. Batkin, Oscillometric blood pressure estimation: past, present, and future, IEEE reviews in biomedical engineering 8 (2015) 44–63. [18] D. Castaneda, A. Esparza, M. Ghamari, C. Soltanpur, H. Nazeran, A review on wearable photoplethysmography sensors and their potential future applications in health care, International journal of biosensors & bioelectronics 4 (2018) 195. [19] M. Panwar, A. Gautam, D. Biswas, A. Acharyya, Pp-net: A deep learning framework for ppg-based blood pressure and heart rate estimation, IEEE Sensors Journal 20 (2020) 10000–10011. [20] Y. Lu, C. Wang, M. Q.-H. Meng, Video-based contactless blood pressure estimation: A review, in: 2020 IEEE International Conference on Real-time Computing and Robotics (RCAR), IEEE, 2020, pp. 62–67. [21] F. Schrumpf, P. Frenzel, C. Aust, G. Osterhof, M. Fuchs, Assessment of deep learning based blood pressure prediction from ppg and rppg signals, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3820–3830. [22] B.-F. Wu, B.-J. Wu, B.-R. Tsai, C.-P. Hsu, A facial-image-based blood pressure measurement system without calibration, IEEE Transactions on Instrumentation and Measurement 71 (2022) 1–13. [23] Y. Chen, J. Zhuang, B. Li, Y. Zhang, X. Zheng, Remote blood pressure estimation via the spatiotemporal mapping of facial videos, Sensors 23 (2023) 2963. [24] X. Niu, S. Shan, H. Han, X. Chen, Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation, IEEE Transactions on Image Processing 29 (2019) 2409–2423. [25] Z. Yu, X. Li, G. Zhao, Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks, in: Proceedings of the British Machine Vision Conference, 2019. [26] M. A. Mehrabadi, S. A. H. Aqajari, A. H. A. Zargari, N. Dutt, A. M. Rahmani, Novel blood pressure waveform reconstruction from photoplethysmography using cycle generative adversarial networks, in: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), IEEE, 2022, pp. 1906–1909. [27] L. N. Harfiya, C.-C. Chang, Y.-H. Li, Continuous blood pressure estimation using exclusively photopletysmography by lstm-based signal-to-signal translation, Sensors 21 (2021) 2952. [28] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017). [29] P.-J. Toye, Vital videos: A dataset of videos with ppg and blood pressure ground truths, arXiv preprint arXiv:2306.11891 (2023). [30] X. Li, I. Alikhani, J. Shi, T. Seppanen, J. Junttila, K. Majamaa-Voltti, M. Tulppo, G. Zhao, The obf database: A large face video database for remote physiological signal measurement and atrial fibrillation detection, in: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE, 2018, pp. 242–249. [31] Z. Yu, W. Peng, X. Li, X. Hong, G. Zhao, Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 151–160. [32] Z. Sun, The 3rd repss track 2, 2024. URL: https://kaggle.com/competitions/the-3rd-repss-t2.