The Video Photoplethysmography (VPPG) technique, while increasingly popular due to its convenience and cost-efectiveness, faces challenges in handling continuous head movements and vigorous motion interferences encountered in real-life scenarios. In this paper, we present a Transformer-based approach aimed at enhancing the robustness of heart rate estimation from facial videos. Leveraging the selfattention mechanism inherent in Transformers, our method adeptly captures both temporal dependencies and spatial information, thereby elevating the accuracy and resilience of heart rate estimation, even in challenging conditions. Through extensive experiments conducted on real-world face video datasets, we illustrate the efectiveness of our approach. Our results demonstrate significant improvements over existing methods in mitigating motion artifacts and enhancing the reliability of non-contact heart rate estimation in practical environments.
The pulse is one of the physiological rhythms in our body, and much information about the state of the body can be obtained by observing its frequency, regularity and intensity. An abnormal pulse may indicate underlying health problems such as arrhythmia, tachycardia or bradycardia. While VPPG technology (video Photoplethysmography) monitors pulse and heart rate through optical sensors [1, 2] and has a wide range of applications including health monitoring devices, clinical diagnostics, exercise physiology and biometrics. Using changes in optical signals, cardiovascular health information is obtained in real time, providing an efective tool for medical treatment, health tracking, and identity verification.Although VPPG technology is widely used, it sufers from shortcomings such as accuracy being afected by the environment, being less precise than ECG, and complex data processing.
Existing research work mainly focuses on the motion interference problem, which can be summarised into the following categories. 1.Optimal facial region selection. The face is divided into several sub-regions, and the optimal detection region is evaluated by analysing the intensity of skin colour changes in diferent regions [ 3] and the degree of influence by motion [4]. This type of method can remove local motion interference such as speech and expression changes, but it is dificult to efectively deal with global motion interference such as head bobbing. 2.Spatial decomposition of pulse representation. Starting from the principle of skin reflection and transmitted light, the decomposition models of pulse signal and motion signal in orthogonal chromaticity space are investigated, including CHROM [5], 2SR [6] and POS [7]. These ideal models have limited ability to cope with the complex mixing of pulse and motion signals. 3.Pulse signal filtering. According to the range and characteristics of human pulse rate variation, band-pass filtering [ 8], wavelet decomposition [9], minimum mean square error filtering [ 10], etc. are used to suppress noise signals other than heartbeat frequency, but it is dificult to separate the interference components with similar frequency characteristics. 4.Blind source separation of pulse signals. According to the time-domain statistical properties of pulse signals, methods such as independent component analysis [11] and sparse representation [12] are used to construct pulse substrates and fit them to reconstruct distorted pulse signals. Due to the limited descriptive ability of such substrates, the separation of motion interference signals is not obvious.
While the application of deep learning methods has become the main direction of current pulse signal extraction research, Contrast-Phys [13] used an unsupervised learning method to generate multiple rPPG signals from diferent spatio-temporal locations in each video using a 3D convolutional neural network (3DCNN) model trained with contrast loss, Contrast-Phys+ [14] used a 3DCNN model to generate multiple spatio-temporal rPPG signals and incorporates a priori knowledge of rPPG into the contrast loss function, Privacy-phys [15] A new approach based on pre-trained 3D convolutional neural networks for modifying rPPG in facial videos for privacy preservation, MTTS-CAN [16] combines a temporal displacement module, an attentional module, and a multitasking mechanism to improve accuracy and eficiency, PhysNet [ 17] uses a deep spatio-temporal convolutional network to recover remote photovoltaic volumetric pulsogram (rPPG) signals from face videos, which can reveal the potential separability of pulse signals from motion signals driven by training data.
The attention mechanism of Transformers excels in handling noisy signals. It enables the network to establish better connections between diferent parts of the signal, efectively distinguishing noise and preserving essential features of the pulse signal.
Face detection and tracking is crucial since slight head movements of the subject are inevitable in practical applications. In the recorded video, the face region is tracked in order to eliminate the rigid motion of the face region. In this paper, we use the facial tracking method introduced in [18]. We used the SDK provided by MediaPipe to implement this facial tracking functionality.
The chromaticity space of the video is converted from RGB space to CHROM space [5] to highlight the colour changes due to impulses. For each pixel, two colour signals were computed X = 3R - 2G and Y = 1.5R + G - 1.5B. The two signals were filtered in a band-pass (0.7-4.0Hz) manne and then combined to form the = − signal, where = () / ( ) and is the standard deviation.
Defining the ROI follows two rules: the first rule is to exclude the eye region because blinking may interfere with the estimated HR frequency; the second is to indent the ROI boundary with the face boundary. Therefore, the cheeks were chosen as the region of interest (ROI), which is less afected by hair and speech. The ROI is labelled in each frame by connecting the four facial marker points around the cheeks with straight lines, where all pixels are globally averaged. Thus, a time series showing changes in skin colour can be constructed.
The time series was linearly interpolated into a 300-element colour signal to achieve signal length consistency. The colour signal is then processed using wavelet decomposition methods to remove noise outside the heart rate band. In this paper, the Meyer wavelet is used to decompose the original colour signal into an approximate component 5 and five detail components 1 ∼ 5, from which the fourth detail component 4 as the colour signal containing the pulse information.
The pre-processed pulse signal contains noise, in order to extract the pulse wave signal accurately we use Transformer network. The function of this network is to receive the preprocessed pulse signal as input and output the pulse wave signal after removing the noise.The Transformer network is able to eficiently capture the long range dependencies in the signal and improve the accuracy and generalisation of the signal extraction, which enables us to analyse the pulse signal more reliably.
Since pulses are periodic and consistent, whereas noisy signals lack such consistent characteristics, Transformer’s attention mechanism can be advantageous when dealing with noisecontaining signals. This mechanism allows the network to better establish connections between diferent parts of the signal, thus efectively distinguishing noise and preserving important features of the pulse signal.
Transformer has achieved remarkable success through its unique self-attention mechanism and positional coding. This model mainly consists of encoder and decoder. In Transformer encoder, multi-head attention and fully connected feed forward network layer are the main components.
The attention mechanism first maps the feature vectors to diferent linear spaces to obtain
three diferent vectors: the Queries vector ( ), the Keys vector (), and the Values vector ( ),
and then obtains the attention vector according to Equation (
MultiHead(, , ) = Concat(head1, · · · , headℎ) 0 head(, , ) = Attention(, , )
T
Attention(, , ) = Softmax( √ ) where denotes the dimension of the vector ().
The multi-head attention mechanism feeds the input vectors to multiple parallel attention
mechanisms for computation, splices the output vectors, and then maps them back to the space
of the original input vectors to obtain the final attention vectors. The specific calculation is
shown in the following equations (
Without introducing an attention mechanism, we first encode the pulse signal to obtain a feature vector and then decode this feature vector to generate the optimised pulse signal.
After introducing the attention mechanism, we first encode the pulse signal to obtain the feature vector . Next, we multiply with the attention score computed from the product of and to obtain the weighted feature vector . Finally, we decode to generate the optimised pulse signal. Since the attention mechanism assigns a higher weight to the segment with periodicity, while the distorted segment is assigned a lower weight , the pulse signal obtained by decoding is usually better than the one obtained by decoding directly.
An interpolation Fourier transform (IFT) [19] is implemented on the reconstructed iPPG signals
to obtain its high-resolution frequency spectrum, from which the average heart rate can be
detected using a peak detecting procedure. This process can be formulated as follows,
Hz = arg max ( )
(
The model was trained on the UBFC-rPPG dataset, which records video at 30 frames per second, 640x480 resolution, in uncompressed 8-bit RGB format, while reference data, such as PPG waveforms and heart rate, were recorded using a CMS50E Transmissive Pulse Oximeter.
For testing, the model was evaluated on two datasets provided by the challenge (OBF and VIPL-HR-V2).The OBF dataset contains 500 videos of 100 subjects, 10 seconds each, with a resolution of 1080p and a frame rate of 30 frames per second, all recorded in static scenes, with the main challenge being the diference in subjects’ skin colour. The VIPL- HR-V2 dataset also contains 500 videos of 100 subjects, each video is 10 seconds long, with a resolution of 720p and a frame rate of 13-25 fps, the videos are recorded in dynamic scenes where subjects perform actions such as talking and shaking their heads. 3.2. Set up This work utilizes a Transformer model for training, with 20% of the training set reserved for validation. The model’s input and output lengths are set to 150, capturing half of the signal spectrum. Each attention head of the Transformer has 15 hidden units, employing a multi-head self-attention mechanism with ReLU activation. Training parameters are optimized using Adam optimizer with an initial learning rate of 0.1 and updated via backpropagation over 100 epochs. Training iterations involve batches of 64 samples. The experimental setup utilizes Tensorflow 2.0 and Matlab 2022b for data processing.
In the RePSS challenge, the performance of the proposed method is evaluated using the Root Mean Square Error (RMSE) as a metric to calculate the RMSE between the ground truth heart rate, , and the measured value, ′.The RMSE reflects the extent to which the measured data is far away from the true value and measures the standard deviation of the residuals. The calculation is shown below: ⎯
= ⎷⎸⎸ 1 ∑=︁1 ( − ′ )2 (5) Where y is the true heart rate value and y’ is the detected value of heart rate.
Table 1 presents the experimental results on the test set provided by the organizer. Our method achieved a Root Mean Square Error (RMSE) of 11.77657 on the test set, which is 22.9% lower than the 8th place. Fig. 2 illustrates the experimental outcomes with six examples. Clearly, the pulse signals, particularly in VIPL-HR-V2, exhibit enhanced regularity post-processing with our method. In the static dataset (OBF dataset), the subject remains stationary, and the raw pulse signals (red curve) show steady fluctuations. Our method accurately captures heart rate variations and extracts periodic heart rate signals from the stable signal. In the dynamic dataset (VIPL-HR-V2 dataset), movements such as head rotation and nodding induce disturbances in the predicted pulse signal, resulting in larger fluctuations in the observed raw pulse signal (red curve). However, our method efectively filters motion interference and accurately extracts the heart rate signal, as evident from the predicted pulse signal (blue curve) in the figure.
To verify the efectiveness of the model proposed in this article, we compared the test results
obtained directly using Fourier transform after preprocessing with the results obtained by
adding the model proposed in this article. As shown in Table 2, the proposed model method
performs better in this experiment, confirming the efectiveness of the model.
3.6. Limitation
(
We propose a method to address the challenges of VPPG in the face of violent motion disturbances using Transformer technology. By feeding the features extracted by the VPPG into the Transformer model for sequence modelling, we are able to capture long distance dependencies between input sequences. This approach promises to suppress motion interference in real-time or ofline scenarios, thereby improving the accuracy and stability of VPPG in detecting impulse signals in face videos. [5] G. De Haan, V. Jeanne, Robust pulse rate from chrominance-based rppg, IEEE transactions on biomedical engineering 60 (2013) 2878–2886. [6] W. Wang, S. Stuijk, G. De Haan, A novel algorithm for remote photoplethysmography: Spatial subspace rotation, IEEE transactions on biomedical engineering 63 (2015) 1974– 1984. [7] W. Wang, A. C. Den Brinker, S. Stuijk, G. De Haan, Algorithmic principles of remote ppg,
IEEE Transactions on Biomedical Engineering 64 (2016) 1479–1491. [8] M.-Z. Poh, D. J. McDuf, R. W. Picard, Advancements in noncontact, multiparameter physiological measurements using a webcam, IEEE transactions on biomedical engineering 58 (2010) 7–11. [9] D. Wang, X. Yang, X. Liu, J. Jing, S. Fang, Detail-preserving pulse wave extraction from facial videos using consumer-level camera, Biomedical optics express 11 (2020) 1876–1891. [10] X. Li, J. Chen, G. Zhao, M. Pietikainen, Remote heart rate measurement from face videos under realistic situations, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 4264–4271. [11] W. Wang, S. Stuijk, G. De Haan, Exploiting spatial redundancy of image sensor for motion robust rppg, IEEE transactions on Biomedical Engineering 62 (2014) 415–425. [12] X. Liu, X. Yang, J. Jin, A. Wong, Detecting pulse wave from unstable facial videos recorded from consumer-level cameras: A disturbance-adaptive orthogonal matching pursuit, IEEE Transactions on Biomedical Engineering 67 (2020) 3352–3362. [13] Z. Sun, X. Li, Contrast-phys: Unsupervised video-based remote physiological measurement via spatiotemporal contrast, in: European Conference on Computer Vision, Springer, 2022, pp. 492–510. [14] Z. Sun, X. Li, Contrast-phys+: Unsupervised and weakly-supervised video-based remote physiological measurement via spatiotemporal contrast, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024). [15] Z. Sun, X. Li, Privacy-phys: Facial video-based physiological modification for privacy protection, IEEE Signal Processing Letters 29 (2022) 1507–1511. [16] X. Liu, J. Fromm, S. Patel, D. McDuf, Multi-task temporal shift attention networks for on-device contactless vitals measurement, Advances in Neural Information Processing Systems 33 (2020) 19400–19411. [17] Z. Yu, X. Li, G. Zhao, Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks, arXiv preprint arXiv:1905.02419 (2019). [18] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee, et al., Mediapipe: A framework for building perception pipelines, arXiv preprint arXiv:1906.08172 (2019). [19] E. Aboutanios, B. Mulgrew, Iterative frequency estimation by interpolation on fourier coeficients, IEEE Transactions on signal processing 53 (2005) 1237–1242. [20] X. Liu, X. Yang, D. Wang, A. Wong, Detecting pulse rates from facial videos recorded in unstable lighting conditions: An adaptive spatiotemporal homomorphic filtering algorithm, IEEE Transactions on Instrumentation and Measurement 70 (2020) 1–15. [21] R. Song, J. Li, M. Wang, J. Cheng, C. Li, X. Chen, Remote photoplethysmography with an eemd-mcca method robust against spatially uneven illuminations, IEEE Sensors Journal 21 (2021) 13484–13494. doi:10.1109/JSEN.2021.3067770.