Introduction

Blood pressure (BP) is an important vital sign in diagnosing certain cardiovascular diseases such as hypertension [1,2,3]. There are two kinds of BP in the human body, namely diastolic blood pressure (DBP) and systolic blood pressure (SBP), which represent the pressure of blood on blood vessels during contraction and relaxation of the heart. In real life, BP is usually measured by contact detection instruments or wearable medical devices. Auscultation is the most traditional method of BP measurement which can well determine the BP state at the time of measurement but it's often influenced by the experience of the auscultator and the environment, resulting in measurement errors. Although cuff oscillography can overcome some shortcomings of auscultation, the inflatable cuff tends to bring uncomfortable experience to the personnel being tested. Therefore, it is of great significance to study convenient and accurate non-contact BP as well as other physiological signals measurement for health monitoring. In order to solve the discomfort and inconvenience caused by contact measuring equipment. Remote photoplethysmography (rPPG) [4,5,6,7,8] methods are developing fast in recent years, which aim to measure heart activity remotely without any contact, makes non-contact physiological signal measurement possible. To study more robust computer vision algorithms and biomedical signal processing methods for extracting physiological signals from facial videos, the 3rd Vision-based Remote Physiological Signal Sensing (RePSS) workshop will be held in conjunction with the International Joint Conference on Artificial Intelligence (IJCAI 2024). There are two tracks in The 3rd RePSS challenge, and the task of the Track2 is facial video-based BP measurement.

BP is closely related to various physiological signals, among which the pulse transit time (PTT) [9] is the most representative. To be specific, PTT refers to the time for a pulse to travel between two different body parts. According to whether PTT is used, contactless BP measurement methods can be divided into PTT-based methods and None-PTT methods. Both types of methods have their limitations, PTT-based methods have high requirements for video frame rate, content and stability. The None-PTT methods are vulnerable to individual differences.

Considering the feasibility of BP measurement based on facial videos, our approach focus on the analysis of physiological signals in the facial video. To be specific, blood volume pulse (BVP), heart rate (HR), body mass index (BMI) and age are extracted from the facial video and frame. Then, RBP-CNN captures the high-dimensional features from BVP to measure the dynamic information in the facial video. Finally, the ensemble algorithm random forest (RF) is used for feature fusion and BP measuring.

Related works

PTT-based methods

Remote BP measurement is sensitive to head shaking [10,11]. Noncontact PTT based BP measurements often require video of multiple parts or other signal support to improve robustness. For example, Fan et al. [12] extract a palm-to-face PTT from the video and feeds it into a physical BP model. Wu et al. [13] employ PTT from two face regions and fuses heart rate variability (HRV), BMI and BVP into a multi-modal model for BP measurement. None-PTT methods Different from the PTT-based methods, the None-PTT methods measure BP by fusing physiological signals such as BVP, HR, HRV, BMI, and age. Zhou et al. [14] input the peaks and troughs of the BVP waveform into a linear regression model to predict BP. Rong et al. [15] extract 26 features from BVP for BP estimation, and train them through four machine learning algorithms. In addition to BVP features, Luo et al. [16] take 29 meta-features(room temperature, subjects'ages, weight, etc.) into account to estimate BP.

Methodology

Our method can be divided into two stages: RBP-CNN training to extract the BP-related feature from BVP and RF training for multi-feature fusion and BP measuring. In this section we will detail each of these stages in turn.

RBP-CNN for BVP feature extraction

As shown in Figure 1, the first stage of our method is BVP feature extraction using RBP-CNN. In this part, we'll introduce in the following order: BVP extraction, principle and structure of RBP-CNN, and loss function.

Nowadays, there are already many excellent unsupervised [17,18,19,20,21] and supervised [22,23,24,25] methods that can represent BVP signals from facial videos. Robust pulse rate from chrominance-based rppg (CHROM) [18] is a traditional and effective unsupervised method, which is used in our BVP extraction. The extracted BVP signal is a one-dimensional time series that changes with time. In previous studies, BVP signal are often used to measure physiological signals such as HR and HRV. However, we believe that in addition to these important reference indicators in medicine, there are high-dimensional features related to BP implied in BVP.

We design RBP-CNN based on residual convolution, local and global attention mechanism to learn features of BVP signals. ResNet [26] has a strong feature representation ability with residual connections and it is widely used in time series analysis. The local attention mechanism is used to focus on local regions within sequence data dynamically and selectively. In the context of time series data, the local attention mechanism enables the model to adjust its attention based on specific parts of the input sequence, allowing for more effective capture of key information within the sequence. The global attention mechanism is employed to consider information from the entire input sequence when making predictions or feature representation. When processing time series data, the global attention mechanism enables the model to weight all time steps equally, allowing it to capture global patterns and relationships within the sequence.

As illustrated in Figure 1, RBP-CNN consists of three 1D residual blocks (depicted in the left part of Figure 1), local attention, global attention and two fully connected layers. We feed BVP signal and BP into RBP-CNN, and BVP is first mapped to high-dimensional feature space through three residual blocks. Then, the weight of each time step of BVP is adjusted and weighed by the local attention and global attention mechanism. Finally, the dimension is reduced by two fully connected layers, and the BP is predicted.

It's worth noting that BP measurement based on multiple physiological signals is essentially an imbalanced regression [27,28,29] task. Because it can be easily observed that most training samples of BP regression concentrate on adults and middle-aged people, while the samples of children and elderly people are fewer, so the labels are imbalanced. To cope with this problem, balanced mean squared error (BMSE) [30] is proposed, which addresses the label imbalance from a statistical perspective, below we give a brief introduction to its principle.

The 𝑦 pred regression can be modeled as a Gaussian distribution, and the mean squared error(MSE) is equivalent to the negative log-likelihood loss of this distribution 𝑝(𝑦 | 𝑥; 𝜃). So, training the MSE regression model is equivalent to modeling the distribution.

𝑝(𝑦 | 𝑥; 𝜃) = 𝒩 (︀ 𝑦; 𝑦 pred , 𝜎 2 noise I )︀ , (1)

where 𝑦 is the label, 𝑥 is the input, 𝜃 is the regressor's parameter, 𝑦 pred is the regressor's prediction and 𝜎 noise is the scale of an i.i.d error term 𝒩 (︀ 0, 𝜎 2 noise I )︀ . In the imbalanced regression task, we train on an imbalanced distribution 𝑝 train (𝑦 | 𝑥) and test on a balanced distribution 𝑝 bal (𝑦 | 𝑥), which leads to a distribution mismatch. By Bayes' rule we get:

𝑝 train (𝑦 | 𝑥) 𝑝 bal (𝑦 | 𝑥) ∝ 𝑝 train (𝑦) 𝑝 bal (𝑦)(2)𝑝 train (𝑦 | 𝑥) = 𝑝 bal (𝑦 | 𝑥) • 𝑝 train (𝑦) ∫︀ 𝑌 𝑝 bal (𝑦 ′ | 𝑥) • 𝑝 train (𝑦 ′ ) 𝑑𝑦 ′ .(3)

Finally, for a regressor's prediction 𝑝 train , and a training label distribution prior 𝑝 train (𝑦 | 𝑥), the BMSE loss is defined as:

𝐿 = − log 𝑝 train (𝑦 | 𝑥; 𝜃) = − log 𝑝 bal (𝑦 | 𝑥; 𝜃) • 𝑝 train (𝑦) ∫︀ 𝑌 𝑝 bal (𝑦 ′ | 𝑥; 𝜃) • 𝑝 train (𝑦 ′ ) 𝑑𝑦 ′ ∼ = − log 𝒩 (︀ 𝑦; 𝑦 pred , 𝜎 2 noise I )︀ + log ∫︁ 𝑌 𝒩 (︀ 𝑦 ′ ; 𝑦 pred , 𝜎 2 noise 𝐼 )︀ • 𝑝 train (︀ 𝑦 ′ )︀ 𝑑𝑦 ′ ,(4)

where ∼ = hides a constant term − log 𝑝 train (𝑦). It can be noted that the calculation of BMSE loss involves the calculation of a double integral. For simplicity, we use its batch-based Monte-Carlo (BMC) approximate implementation for the loss calculation of RBP-CNN.

Multi-feature fusion with Random Forest

As demonstrated in Figure 2, the second stage of our method is multi-feature fusion and BP prediction based on RF. In this part, we will introduce our scheme in the order of feature extraction, feature correlation analysis, and feature fusion. In medicine, the primary cause of hypertension is arteriosclerosis, and the most directly associated factors with arteriosclerosis are age and BMI, that's why hypertension is more prevalent in obese and middle-aged to elderly populations. Additionally, these individuals are also prone to show abnormalities in HR. Therefore, we take care of age, BMI, and HR as important features for BP measurement. Specifically, we input any frame from the facial video into pre-trained models [31] to estimate age and BMI. Heart rate is calculated from BVP signal by fourier transform.

It can be seen from Figure 3 that the DBP's pearson correlation coefficients with age, BMI, HR are 0.301, 0.238 and 0.133 respectively (p<0.001) among which DBP is moderately correlated with age and weakly correlated with BMI and HR. As shown in Figure 4, the pearson correlation coefficients of SBP with age, BMI and DBP are 0.562, 0.286 (p<0.001) and 0.704 (p<0.05), indicating that they have moderate, weak and strong correlations with SBP respectively. Commonly used physiological information such as age, BMI and HR have been used in the previous works. Researchers have focused on the relationship between them and BP, however, few people pay attention to the internal correlation between DBP and SBP. We notice it and utilise DBP in SBP prediction.

Multi-feature fusion is realized by RF [32], which is a widely used ensemble algorithm. RF is composed of multiple decision trees, and the final prediction result is determined by the voting results of each decision tree. In regression task, the output of each decision tree is a continuous value, and the average of the output results of all decision trees is taken as the final result. RF can deal with high-dimensional and imbalanced datasets, and has the advantages of high accuracy and robustness. At the same time, we can also evaluate the importance of features, which is helpful for us to verify the effectiveness of features through experiments.

Experiments

In the 3rd RePSS challenge Track2, we use 322 samples from Vital Videos for training, of which 162 samples are used for training RBP-CNN and 160 for training the random forest. 200 label unknowned samples from OBF are used for testing.

Datasets

Vital Videos (VV) [33] is a public dataset of videos with PPG and BP ground truths, which in total contains information about 900 different participants. For each participant, 2 or 3 30s uncompressed video are collected, along with personal information (gender, age, skin color), PPG, HR, blood oxygen saturation and BP. The dataset includes roughly equal numbers of males and females, as well as participants of all ages, skin color in different locations, ensuring a variety of different background and lighting conditions.

OBF [34] is a large face video database for remote physiological signal measurement and atrial fibrillation (AF) detection. It contains data from 100 healthy individuals as well as six AF patients. For each participant, multi-modal data (RGB videos, NIR video, ECG, BVP, RF) are recorded simultaneously during two phases, each lasting 5 minutes. For healthy participants and patients with AF, the first and second phases are resting state, post-exercise HR increase, before and after cardioversion treatment respectively.

Training Procedure

For the RBP-CNN training stage, the BVP is extracted from the corresponding facial video, after which the HR is calculated. Subsequently, the BVP signals and BP labels are fed into RBP-CNN for training. After that, the last fully connected layer of RBP-CNN is removed and it becomes a BVP feature extractor. We then feed BVP into the feature extractor to capture BP-related features. At last, both BVP feature and other physiological information are used to build the RF training set, and then fit the DBP and SBP through the RF regression model.

For the RF regressor training stage, two regressors (DBP regressor and SBP regressor) are trained. The BVP and HR are calculated from the facial video. At the same time, the first frame of each sample's facial video is used for BMI estimation (age is available in VV). Lastly, the features learned from BVP, age, BMI, HR, DBP (SBP regressor training only) are fused to train DBP and SBP RF regressors.

There are 507 samples from 250 participants for training initially. To improve performance across datasets, we use the evaluated age to approximate the distribution of the training set to the test set distribution. Finally, 322 samples from 160 participants are selected, among which 162 and 160 samples are used for training RBP-CNN and RF regressor.

The RBP-CNN model is implemented based on pytorch framework and trained on a NIVIDA GeForce GTX 1650 GPU for 200 epochs with a learning rate of 0.001. The RF regressor is implemented based on sklearn and trained on lntel (R) Core (TM) i5-9300H CPU and the n_estimators, max_depth, criterion are set to 1000, 6, absolute_error respectively.

Evaluation Metric

In the training process of RBP-CNN model, we use mean absolute error (MAE) and MSE as the model evaluation metric. For the actual test of Track 2, The root mean squared error (RMSE) of the ground truth DBP, SBP with the submitted ones are calculated successively, and then they are averaged as the final race score.

𝑅𝑀 𝑆𝐸 2 = 0.5 √︃ ∑︀ 𝑁 𝑖=1 (𝑠 𝑖 − 𝑠 ′ 𝑖 ) 2 𝑁 + 0.5 √︃ ∑︀ 𝑁 𝑖=1 (𝑑 𝑖 − 𝑑 ′ 𝑖 ) 2 𝑁(5)

where 𝑠 𝑖 is the ground truth SBP of the ith test sample, 𝑠 ′ 𝑖 is the submitted SBP of the ith test sample. Similarly, 𝑑 𝑖 and 𝑑 ′ 𝑖 are the ground truth DBP and submitted one of the ith test sample.

Results

As shown in Table 1, our team (PCA_Vital) achieves second place on the 3nd RePSS Challenge Track 2. The final score of our submitted BP prediction is 13.48281, which is behind first place 0.53023 mmHg and ahead of the third place 0.11026 mmHg but significantly ahead of the fourth place.

Figure 5 shows the top 20 feature importance of DBP and SBP RF regressors. It can be observed that for both DBP and SBP regressors, BMI and age rank among the top three feature importance. Notably, the feature importance of DBP in SBP RF regressor comes to 0.52, which is obviously ahead of other features. It confirms the strong correlation between DBP and SBP. At the same time, BVP features also have a great contribution in BP measurement. Through calculation, the cumulative importance of BVP features reaches 0.56 and 0.30 in the DBP and SBP RF regressors, respectively, verifying the effectiveness of RBP-CNN based BVP feature extraction.

Conclusion

This paper presents a video-based remote BP measurement scheme via convolutional network and RF feature fusion. We combine residual convolution, local and global attention mechanisms to design RBP-CNN for learning the implicit BP-related information in BVP spatially and temporally. Subsequently, we capture BMI, age, HR from facial video and analyze their correlation with BP. In this process, we find a strong correlation between DBP and SBP. At last, we use RF to fuse these features to achieve BP measurement and verify the rationality of our method by using the feature importance of RF. Our method achieves second place on the 3nd RePSS Challenge Track 2, and we believe we can do better in the future.