This paper briefly introduces the solutions developed by our team, HFUT-VUT, for Track 1 of selfsupervised heart rate measurement in the 3rd Vision-based Remote Physiological Signal Sensing (RePSS) Challenge hosted at IJCAI 2024. The goal is to develop a self-supervised learning algorithm for heart rate (HR) estimation using unlabeled facial videos. To tackle this task, we present two self-supervised HR estimation solutions that integrate spatial-temporal modeling and contrastive learning, respectively. Specifically, we first propose a non-end-to-end self-supervised HR measurement framework based on spatial-temporal modeling, which can efectively capture subtle rPPG clues and leverage the inherent bandwidth and periodicity characteristics of rPPG to constrain the model. Meanwhile, we employ an excellent end-to-end solution based on contrastive learning, aiming to generalize across diferent scenarios from complementary perspectives. Finally, we combine the strengths of the above solutions through an ensemble strategy to generate the final predictions, leading to a more accurate HR estimation. As a result, our solutions achieved a remarkable RMSE score of 8.85277 on the test dataset, securing 2nd place in Track 1 of the challenge.
CEUR ceur-ws.org
Remote physiological measurement [
Despite the promising potential of video-based physiological measurement, most existing
methods [
To address these challenges, the 3rd Vision-based Remote Physiological Signal Sensing
(RePSS) Challenge at IJCAI 2024 was launched. This challenge aims to develop self-supervised
training methods for HR measurement using unlabeled facial videos, thereby reducing the
dependency on extensive labeled datasets. For this challenge, we present two self-supervised
HR estimation solutions that integrate spatial-temporal modeling and contrastive learning,
respectively. Inspired by Dual-TL [
In conclusion, the main contributions can be summarized as follows: • We propose a non-end-to-end self-supervised solution based on spatial-temporal modeling.
By considering the priors of periodicity consistency and bandwidth limitation of the rPPG signal, we introduce four loss functions to supervise the model efectively. • We present an end-to-end solution based on contrastive learning, which utilizes 3DCNN to extract features and employs a contrastive loss to learn discriminative representations for periodic rPPG signal modeling. • Our solution achieved second place with the RMSE score of 8.85277 on the test dataset in Track 1 of the 3rd Vision-based Remote Physiological Signal Sensing Challenge. The experimental results demonstrate the efectiveness and robustness of our proposed solutions.
2.1. Solution 1: Self-supervised HR Measurement with Spatial-Temporal
Transformer Inspired by the great success of Transformer in computer vision [15], we present a non-endto-end self-supervised HR measurement framework to mitigate the need for labeled video N ROI Combinations
… … ROI-1 AvReOraI-g3e6pooRliOngI-N ×× isng ∈ℝ d d e b m E
Spatial-Temporal Transformer Spatial Encoder Temporal Encoder
× reryaoLNm iltaaSp li-fttteeonSnA+ m r o rN PL+ eya M L errayoLNm leroapTm li-fttteeonnSA+ m r o rN PL + eya M L ×ℝ ∈ ion s rs e egrPPG signal R ℒ = ℒ)*++ ℒ+ ℒ+ ℒ
SPD variance loss ℒ bandwidth loss ℒ sparsity loss ℒ 0 0.66 3.0 HZ clip A clip B clip C PSD PSD PSD periodicity loss ℒ data based on a Spatial-Temporal Transformer. The overview of this solution is illustrated in Figure 1. Specifically, we first transform the input facial video into a multi-scale spatial-temporal map (MSTmap) in Section 2.1.1. Then, we introduce our spatial-temporal Transformer module in Section 2.1.2. Next, in Section 2.1.3, with the constraints of periodicity consistency and bandwidth finiteness, our model directly discovers blood volume pulses from unlabeled videos to predict HR.
The quasi-periodic pulse signal originates from subtle light reflections of blood vessels under the skin. Therefore, non-skin pixels and facial geometric features can be considered as rPPG-independent noises. We transform the raw facial video into MSTmap to highlight the spatiotemporal information of the human face, which is a common practice in rPPG measurement [16, 17]. Concretely, the MSTmap divides the facial area into 6 meta-ROI blocks, which can generate = (26-1)=63 ROI combination blocks, and the pixels of each block are averaged separately for color channels. In the video, all the frames are concatenated along the time dimension to generate a spatial-temporal map of size ℝ × × , where = 6 represents {R,G,B,Y,U,V} channels. Next, we embed the MSTmap to high-dimensional feature ∈ ℝ × × with feature dimension by using a full-connected layer.
Our spatial-temporal Transformer tailored for remote physiological measurement is designed carefully for perceiving the temporal and spatial correlations. It includes two encoders (spatial encoder and temporal encoder) to refine the ROI representation containing rPPG clues by capturing long-term spatiotemporal contextual information. We now explain the proposed model in detail. Specifically, given the input features ∈ ℝ × × , the process of embedding spatial context for -frame can be formulated as: () = () , () = () , () = () , () √
() () = softmax (
) () + () , ′() = MLP(LN( () )) + () , (1) (2) where , , are learnable parameters shaped as × frame. MLP is the multi-layer perceptron layer and LN is layer normalization operation. The . () denote the feature in -th feature map of all frames { ′() | = 1, … , } are concatenated together into ∈ ℝ × × We output the temporally correlated feature for the -th facial ROI feature as ′() ∈ ℝ × Eq. 1. The diference is that we calculate the temporal dimension for each spatial unit ( ∈ [1, ] ). and stack the features { ′() | = 1, 2, … , } together, represented by ∈ ℝ × ×
The spatial and temporal encoders are stacked as loops in an alternating manner, taking into account the spatial and temporal complementary contextual information integrally. Moreover, spatial and temporal position embedding is applied only to the first encoder to retain two kinds of position information. Finally, we use an rPPG regression head to project the feature to a 1D rPPG signal
∈ ℝ ×1 .
As highlighted in previous studies [18, 14], the rPPG signal possesses inherent theoretical priors, including specific bandwidth in the frequency domain. By incorporating this prior knowledge, we employ three self-supervised loss functions from [14] in this work. Additionally, to further efectively train the model, we also propose a new periodicity loss based on periodic characteristics of the rPPG signal. Notably, all predicted rPPG signals are transformed into power spectrum density (PSD) with the Fast Fourier Transform (FFT) before computing all losses in our method, denoted as =
FFT( ).
Bandwidth Loss. A healthy HR falls within a specific frequency range. Following the [ 14], we penalize the model for producing signals that exceed the healthy HR bandwidth limits. Consequently, the bandwidth loss can be formalized as follows:
L = [ ∑ + ∑ ] ,
1 ∞ =−∞ ∑ =−∞ ∞ = where and denote lower and upper band limits, respectively. is the power in the th frequency bin of the predicted signal. In our experiments, we specify the limits as = 0. 66 Hz to = 3 Hz, which corresponds to a common pulse rate range from 40 bpm to 180 bpm. This range efectively captures the typical variations in a healthy HR, ensuring that our model focuses on the relevant frequency components while minimizing the influence of noise. By incorporating this bandwidth loss, our model is better equipped to distinguish between meaningful rPPG signals and disturbances, ultimately leading to more accurate HR estimation.
Sparsity Loss. Since we are primarily interested in heartbeat frequency, we emphasize the periodic heartbeats by suppressing non-heartbeat frequencies. Following [14], we penalize the energy in the bandwidth regions far away from the spectrum peak, which can ensure that the model focuses on the relevant heartbeat frequencies. It can be formulated as: L =
1 ∑ = [ argmax( )−Δ ∑ = +
∑ = argmax( )+Δ ] , where argmax( ) is the frequency of the spectral peak, and Δ = 6 is the frequency padding around the peak. This loss enhances the model’s ability to accurately estimate HR by ensuring that the spectral energy is concentrated around the true HR frequencies, thus minimizing the influence of noise and other non-relevant frequency components.
Variance Loss. To avoid the model collapsing to a specific frequency, we also use a variance loss [14, 19] to spread the variance of the power spectral density into a uniform distribution over the desired frequency band. Firstly, we define a uniform prior distribution over frequencies. Then, we consider a batch of spectral densities, represented as = [ 1, … , ], where each is a -dimensional frequency decomposition of a predicted waveform. To aggregate these spectral densities, we compute the normalized sum across the batch, denoted as . Therefore, the variance loss L can be formulated as:
1 =1 L =
∑ (CDF () − CDF ( ) )2 , where CDF represents the cumulative distribution function at the -th frequency.
Periodicity Loss. In addition to the intrinsic properties of the rPPG signal itself, we have observed that adjacent rPPG signals do not change rapidly over short periods. This is typically manifested by similar periodicity in neighboring rPPG signals, meaning they share a dominant peak in the PSD. Specifically, we uniformly sample non-overlapping temporal segments from a short rPPG signal (e.g., 10s). The PSDs of these segments should be similar. Thus, our proposed periodicity loss can be formulated as:
L
−1 = ∑ ∞ ∑ ( −
=1 =−∞ +1 )2 , where = 3 denotes the number of segments.
In summary, the overall loss function of our self-supervised learning strategy is : L = L + L + L + L .
(3) (4) (5) (6) Video 1
3DCNN Video 2 r e l p m a s T S
3DCNN Video
2.2. Solution 2: Self-supervised HR Measurement with Contrastive Learning Here we provide the end-to-end self-supervised HR measurement framework based on the contrastive learning strategy. The framework is depicted in Figure 2. Specifically, we first perform data-preprocessing in Section2.2.1. Then we pre-train the proposed model in an unsupervised setting based on the Contrast-Phys+ [11] in Section 2.2.2. Finally, we fine-tune the Contrast-Phys+ model with a supervised setting and obtain the final rPPG predictor in Section 2.2.3.
In this self-supervised manner, we input facial video into our model to estimate the final rPPG signal. For an original video, we first perform face detection by MTCNN [ 20] to get the four coordinates of the face bounding box from the first frame. Then, we enlarge the length and width of the bounding box by 1.5 times and crop the face region for each frame of the video. The cropped faces are resized to 128 × 128. Next, we segment each video into clips to feed into the model. Note that we also perform frame diference operations on the clip to generate normalized diference frames as an attempt of model input. The diference between two consecutive frames can be formulated as: Δ = +1 − , (7) where denotes the -th frame. To keep the length of the diference video equal to the raw video, we simply repeat the last diference frame. Then, the Δ is normalized.
In this stage, following the setting of [11] we modify the 3DCNN-based PhysNet to get spatiotemporal rPPG (ST-rPPG) block representation. The model outputs spatiotemporal rPPG features with shape × × , where is the temporal length, and is the spatial dimension. The ST-rPPG block can be regarded as a collection of rPPG signals from diferent facial regions. Therefore, for each input, we can sample 2 rPPG signals with the length of .
According to the observations that rPPG spatial similarity and temporal similarity in [11], the ST-rPPG block can sample multiple rPPG signals with short time intervals and diferent spatial positions. Those signals should be similar. Then contrastive learning can be formulated by pulling together the rPPG signals from the same ST-rPPG block and pushing away the signals from diferent ST-rPPG blocks extracted in the crossing video. The contrastive loss can be formulated as:
L L
and the ground-truth PPG signal , a popular Negative Pearson correlation (Pear) loss and Negative max cross-correlation (MCC) loss are selected to perform supervised training. It is worth noting that the Pear is the time domain loss function while the MCC loss is the frequency domain loss function. The MCC is robust to temporal ofsets in the ground truth, which can make up for the Pear loss. The MCC loss is formulated as: where is the PSDs of the ground-truth signal. where denotes the Power Spectrum Densities (PSDs) of the rPPG signal in position and ′ is the other video’s PSDs. is the number of sampled rPPG pairs. The contrastive loss function minimizes the MSE distance between positive samples and maximizes the distance between the negative samples to force the model to learn the discriminative representation of the underlying signals from diferent videos. 2.2.3. Fine-tuning =1 =1 ≠ =1 =1
∑ (‖ − ‖2 + ‖ ′ − ′‖2) /(2 ( − 1)), = − ∑
∑ ‖ − ′‖2 / 2, L = − Max ( −1{ ( {
× } ⋅ { }) ) , where
−1 is the inverse of fast Fourier transform (FFT), is the standard deviation. Besides, as the ground-truth signals are the reference of predicted rPPG signals, the similar to . Therefore, we also use the contrastive loss by the following: should be = ∑
∑ (‖ − ‖2 + ‖ ′ − ′‖2) /(2 ( − 1)), = − ∑ ∑ (‖ − ′‖2 + ‖ ′ − ‖2) / 2, (8) (9) (10) (11) (12) (13)
Finally, the overall loss for fine-tuning is the combination of Pear loss, MCC loss, and contrastive loss, which can resist noise interference of ground-truth signal.
L = L
+ L + L + L , (14) where L is the Negative Pearson correlation loss function. In our experiments, we set to 0.1 and to 0.2 for the VIPL-V2 dataset.
3.1. Datasets
UBFC-rPPG [21] is a commonly used pure dataset for physiological estimation. It records 42
facial videos from 42 subjects in a stable lab environment. PURE [22] contains 60 facial videos of
10 participants under 6 modes (steady, small rotation, medium rotation, talking, slow translation,
and fast translation). MMSE-HR [23] contains 102 facial videos captured from 40 subjects
under six task modes. This dataset contains various facial expression changes. DISFA [24] is a
non-posed facial expression dataset. It records 27 facial videos from 27 subjects with diferent
ethnicities[25]. VIPL-V2 [26] is the second version of the VIPL-HR [26] dataset for remote
HR estimation from face videos under less-constrained situations, which contains 2,000 RGB
videos provided in this challenge [16, 17]. Up until the publication of the OBF [
For solution 1 introduced in Section 2.1, we begin by extracting the facial ROI regions using
the landmark detection tool of OpenFace during the data pre-processing step. We then follow
the setting described in [17], applying a sliding window size of 300 frames (10s) and a step
size of 15 frames (0.5s) to generate MSTmap from the facial videos. For the spatial-temporal
Transformer module, we set the dimensionality to 128 and the number of layers to 6. During
pre-training, we use the AdamW optimizer with a learning rate of 1e-4 and a batch size of
4. Data augmentation techniques include random horizontal and vertical flipping as well as
frequency up/down sampling are used. In the fine-tuning step with data labels, in addition
to the self-supervised loss, we also add Negative Pearson Loss to further optimize the model.
Besides, we use a smaller learning rate, i.e., 1e-5, to finetune the model. For the VIPL-V2 dataset,
we split the training and validation subsets in a ratio of 8:2. For the HR estimation inference
step, following previous work [
For the ensemble strategy, we take the multiple best prediction results under diferent settings of both solution 1 and solution 2. Then we average the diferent predicted heart rates of each sample as the final result. 3.3. Experimental Results Results for Solution 1. As shown in Table 1, we investigate the impact of diferent pretraining datasets and loss functions for solution 1. The results indicate that as the amount of pre-training data increases, the performance of the model improves accordingly. In our solution, we ultimately select the UBFC-rPPG [21], PURE [22], and MMSE-HR [23] datasets for pre-training. Additionally, we also investigate the impact of the proposed periodicity loss L . We can see that the incorporation of the periodicity loss consistently improves the performance of the model significantly across diferent settings. For instance, when the model is trained on the UBFC-rPPG, PURE, and MMSE-HR datasets, the introduction of the periodicity loss reduces RMSE from 10.35720 to 9.93125. This improvement underscores the efectiveness of the periodicity loss in mitigating abnormal periodic fluctuations in the predicted signal and maintaining temporal periodicity consistency.
Results for Solution 2. As shown in Table 2, we evaluate diferent pre-training datasets, loss functions, and model inputs to find the best setting for this task. Note that the DISFA dataset is a non-posed facial expression database. However, from the results, we can find that using it for pre-training can still achieve comparable performance. Apart from that, we can ifnd the same conclusion as solution 1 that increasing the amount of pre-training datasets is beneficial to performance. In this solution, we choose DISFA, UBFC-rPPG, MMSE-HR, and PURE for pre-training. Additionally, we also evaluate diferent combinations of supervised 1https://www.kaggle.com/competitions/the-3rd-repss-t1/leaderboard loss L . The results show that both the time domain and frequency domain loss are helpful for model fine-tuning. Moreover, we evaluate the performance of normalized frame diference input, and it shows a comparable result with normal input. In the model ensemble phase, we added the frame diference-based manner as diferent feature forms.
Model Ensemble. In order to combine the advantages of Solution 1 and Solution 2, we use an ensemble strategy to integrate the best prediction results of these two solutions together. Specifically, we ensembled the models by taking the average value of the prediction results for Solution 1 and Solution 2, and then obtained the final prediction results. As shown in Table 3, we report the top-3 results on the test dataset for each RePSS challenge. Compared to other teams, we can see that our team achieves 2nd place, which is higher than the 3rd by 1.2%. This demonstrates that our proposed two self-supervision solutions can complementaryly achieve more accurate and robust heart rate estimation. Compared to the results of the supervised methods in previous challenges, we can find that self-supervised methods improve performance by a large margin. This indicates that self-supervised methods can capture rPPG-related signals from facial videos during the pre-train phase without requiring any real physiological signals.
In this paper, we present our solutions developed for self-supervised remote heart rate measurement of the 3rd RePSS challenge hosted at IJCAI 2024. Specifically, we propose two selfsupervised HR estimation solutions that integrate spatial-temporal modeling and contrastive learning, respectively. By leveraging the ensemble strategy, our final submission takes second place with the RMSE score of 8.85277 bpm. In the future, we plan to address the issues in this challenge from other perspectives, e.g., using video motion magnification algorithms [ 30] to capture the subtle change reflected in faces by heartbeats.
This work was supported by the National Key R&D Program of China (NO.2022YFB4500601), the National Natural Science Foundation of China (72188101,62272144,62020106007and U20A20183), the Major Project of Anhui Province(202203a05020011), and the Fundamental Research Funds for the Central Universities. [8] K. Li, D. Guo, M. Wang, Vigt: proposal-free video grounding with a learnable token in the transformer, Science China Information Sciences 66 (2023) 202102. [9] D. Guo, K. Li, B. Hu, Y. Zhang, M. Wang, Benchmarking micro-action recognition: Dataset, methods, and applications, IEEE Transactions on Circuits and Systems for Video Technology (2024). [10] Y. Wei, Z. Zhang, Y. Wang, M. Xu, Y. Yang, S. Yan, M. Wang, Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking, IEEE Transactions on Image Processing 30 (2021) 4788–4801. [11] Z. Sun, X. Li, Contrast-phys+: Unsupervised and weakly-supervised video-based remote physiological measurement via spatiotemporal contrast, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024) 1–18. [12] H. Lu, H. Han, S. K. Zhou, Dual-gan: Joint bvp and noise modeling for remote physiological measurement, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12404–12413. [13] Z. Yu, W. Peng, X. Li, X. Hong, G. Zhao, Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 151–160. [14] J. Speth, N. Vance, P. Flynn, A. Czajka, Non-contrastive unsupervised learning of physiological signals from video, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14464–14474. [15] K. Li, J. Li, D. Guo, X. Yang, M. Wang, Transformer-based visual grounding with crossmodality interaction, ACM Transactions on Multimedia Computing, Communications and Applications 19 (2023) 1–19. [16] X. Niu, S. Shan, H. Han, X. Chen, Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation, IEEE Transactions on Image Processing 29 (2019) 2409–2423. [17] X. Niu, Z. Yu, H. Han, X. Li, S. Shan, G. Zhao, Video-based remote physiological measurement via cross-verified feature disentangling, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, 2020, pp. 295–310. [18] J. Gideon, S. Stent, The way to my heart is through contrastive learning: Remote photoplethysmography from unlabelled video, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3995–4004. [19] A. Bardes, J. Ponce, Y. Lecun, Vicreg: Variance-invariance-covariance regularization for self-supervised learning, in: International Conference on Learning Representations, 2022. [20] K. Zhang, Z. Zhang, Z. Li, Y. Qiao, Joint face detection and alignment using multitask cascaded convolutional networks, IEEE Signal Processing Letters 23 (2016) 1499–1503. [21] S. Bobbia, R. Macwan, Y. Benezeth, A. Mansouri, J. Dubois, Unsupervised skin tissue segmentation for remote photoplethysmography, Pattern Recognition Letters 124 (2019) 82–90. [22] R. Stricker, S. Müller, H.-M. Gross, Non-contact video-based pulse rate measurement on a mobile service robot, in: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, 2014, pp. 1056–1062. [23] S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, N. Sebe, Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 2396–2404. [24] S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, J. F. Cohn, Disfa: A spontaneous facial action intensity database, IEEE Transactions on Afective Computing 4 (2013) 151–160. [25] S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, Automatic detection of non-posed facial action units, in: 2012 19th IEEE International Conference on Image Processing, 2012, pp. 1817–1820. [26] X. Niu, H. Han, S. Shan, X. Chen, Vipl-hr: A multi-modal database for pulse estimation from less-constrained face video, in: Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part V 14, 2019, pp. 562–576. [27] P. Welch, The use of fast fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms, IEEE Transactions on Audio and Electroacoustics 15 (1967) 70–73. [28] X. Li, H. Han, H. Lu, X. Niu, Z. Yu, A. Dantcheva, G. Zhao, S. Shan, The 1st challenge on remote physiological signal sensing (repss), in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 314–315. [29] X. Li, H. Sun, Z. Sun, H. Han, A. Dantcheva, S. Shan, G. Zhao, The 2nd challenge on remote physiological signal sensing (repss), in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2404–2413. [30] F. Wang, D. Guo, K. Li, M. Wang, Eulermormer: Robust eulerian motion magnification via dynamic filtering within transformer, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 5345–5353.