With the advancement of remote photoplethysmography (rPPG), video-based physiological signal measurement has emerged as a convenient and contactless method for heart rate (HR) estimation. Most existing video-based rPPG methods rely on either RGB or NIR imaging, each with distinct advantages and limitations. RGB-based methods demonstrate higher accuracy but are sensitive to lighting and skin tone, while NIR-based methods are more robust to such variations but less responsive to blood volume changes, leading to lower accuracy. Therefore, we propose a temporal diference transformer (TDT)-based multimodal fusion framework for robust HR estimation. We employ separate 3D convolutional encoders to independently extract modality-specific spatio-temporal representations from RGB and NIR input streams. Subsequently, the TDT module enhances quasi-periodic features while adaptively aligning and fusing RGB and NIR representations. Additionally, a composite loss function is introduced to provide simultaneous supervision across temporal and spectral domains. In the 4th RePSS Challenge, the proposed method achieved second place with the root mean square error (RMSE) of 12.32 bpm on the oficial test set, demonstrating strong performance in multimodal HR estimation.
Heart rate (HR) is a crucial physiological indicator for assessing health and emotional states [
Despite the success of RGB-based deep learning methods for HR estimation, they still face considerable
challenges in real-world scenarios. In low-light conditions, limited visible light reduces the
signalto-noise ratio of rPPG signals [
× N v n oC BN eLRU + D 3 v n oC BN eLRU + D 3 × N
P M D 3 P M D 3 t a c n o C d e b m E h c t a P m r o N r e y a L
Temporal Difference Transormer× M
n ce itno freen ftt-ea if l lD eS rpoa ehad eTm lit-u
M + m r o N r e y a L l ropa rad itt-epaoSm f-rodeeFw +
( , σ²) n o iit c d e r P KL DiLvoesrsgence Cross-Entropy
PSD LosPsSD Negative Pearson Correlation Loss Reference PPG
To address these limitations, integrating RGB and NIR modalities shows promise for achieving a balance between illumination robustness and sensitivity to blood flow signals. By leveraging the complementary strengths of both modalities, multimodal fusion has the potential to improve the accuracy and generalizability of rPPG estimation in diverse and complex environments. To explore the potential of multimodal data fusion in rPPG, the 4th Remote Physiological Signal Sensing (RePSS) Challenge was launched in conjunction with the International Joint Conference on Artificial Intelligence (IJCAI 2025).
In this challenge, we propose an end-to-end multimodal fusion method based on the temporal
diference transformer (TDT) [
As shown in Fig. 1, our method consists of three main components: modality-specific spatio-temporal encoding, multimodal feature fusion, and rPPG prediction. Furthermore, a composite loss function is designed to provide supervision in both the time and frequency domains. In the following sections, we provide a detailed description of each component.
Given a sequence of facial video frames from two modalities (RGB denoted as XRGB and NIR as XNIR), each modality is independently processed through a 3D convolutional encoder to extract spatio-temporal features specific to that modality. The RGB input has three channels, while the NIR input has one channel, with both sharing the same temporal and spatial dimensions.
Each encoder is composed of multiple stacked blocks, where each block includes a 3D convolutional layer followed by batch normalization, ReLU activation, and 3D max pooling. To improve training stability and maintain feature quality, residual connections are introduced within each block by adding the input features to the output of the transformation.
Through this encoding process, we obtain modality-specific feature representations: FRGB ∈ R× 1× 1× 1 and FNIR ∈ R× 1× 1× 1 , both sharing the same shape in terms of temporal and spatial resolution after downsampling, but carrying distinct semantic information from their respective inputs.
In temporal diference multi-head self-attention (TD-MHSA) [
TDC() = ∑︁ () · (0 + ) + · (︀ − (0) · ∈ℛ ∑︁ ()︀) ∈ℛ′ (1) (2) (3) (4) where 0 denotes the current spatio-temporal location, ℛ is the local 3 × 3 × 3 receptive field, and ℛ′ represents the adjacent temporal neighborhood.
For each attention head , we compute the query/key/value matrices by:
= ︀( TDC (LN()) )︀ , = ︀( TDC (LN()) )︀ , = ︀( Conv3D (LN()) )︀ , where LN(· ) is layer normalization, (· ) denotes the reshaping operation from video to sequence format, and , , ∈ R(2· 2· 2)× 2. This design explicitly injects temporal change patterns into the attention mechanism through diferential operators.
The self-attention for each head is then computed using the scaled dot-product:
SA = softmax ︂( )︂ √ℎ , where ℎ = 2ℎ is the dimensionality of each attention head, ℎ represents the number of attention heads.
The outputs from all ℎ heads are concatenated and linearly projected, then reshaped back to the original video format via (· ), yielding the final TD-MHSA output:
TD-MHSA = (︀ Concat(SA1, SA2, . . . , SAℎ) · )︀ , where is a learnable output projection matrix, and (· ) is the inverse reshape operator (sequence to video).
To enhance local feature consistency and positional awareness, a spatio-temporal feed-forward (ST-FF) module is employed in place of standard linear layers. The module consists of a 1 × 1 × 1 pointwise convolution for channel expansion, a 3 × 3 × 3 depthwise-separable 3D convolution to model spatio-temporal interactions, and a second 1 × 1 × 1 pointwise convolution for dimensionality reduction. Each stage is followed by batch normalization and ELU activation to promote training stability and non-linearity.
After the TDT module completes modality fusion and temporal modeling, the model employs a two-stage upsampling process to progressively restore temporal resolution while reducing channel dimensionality to extract key sequential features. Spatial average pooling is then applied to integrate local information and produce a more compact temporal representation. Finally, a regression module maps the temporal features into a rPPG signal sequence.
To obtain the final HR, the fast Fourier transform (FFT) is applied to the predicted rPPG signal and identify the dominant frequency component within the physiological range (0.7–4 Hz). The frequency with the highest amplitude is selected as the pulse frequency peak.
To enable the model to efectively reconstruct the rPPG waveform and accurately estimate HR, we design a composite loss function consisting of three components, providing supervision from both the time and frequency domains.
KL Divergence Loss [
︂( exp − ( − (HRgt − HRmin))2 )︂ , where HRmin denotes the theoretical minimum HR. Let ˆ represent the softmax-normalized result of the power spectral density (PSD) of the predicted rPPG signal. The KL divergence loss is defined as:
. ℒCE = − log
exp(ˆ) ∑︀=−01 exp(ˆ) )︃
. ℒPearson = 1 −
Cov(, ) ,
Cross-Entropy Loss [
Negative Pearson Correlation Loss [
Overall Loss: The final training objective combines the three losses with weighting coeficients: ℒtotal = · ℒ Pearson + · (ℒKL + ℒCE), where and control the relative importance of the temporal and frequency-domain losses.
The training data comes from the VIPL-HR [
The test set consists of paired RGB-NIR clips from 100 subjects in the VIPL-V2 [
The model proposed in this study is implemented using the PyTorch framework and trained on a high-performance computing system equipped with eight NVIDIA GeForce RTX 4090 GPUs. Each input sample consists of a facial video segment containing 160 frames, with each frame resized to a resolution halves the learning rate every 25 epochs. of 128 ×
128 pixels. During training, the batch size is set to 8, and the total number of training epochs is 50. The model is optimized using the Adam optimizer, with a weight decay coeficient of The initial learning rate is set to 1 × 10− 4, and a StepLR learning rate scheduler is employed, which 5 × 10− 5. where denotes the number of video segments, HRgt represents the ground truth HR of the -th video segment, and HRpred denotes the predicted HR of the -th video segment.
As shown in Table 1, our team, IST (Nanjing University), achieved the RMSE of 12.31846 bpm in the HR estimation task, ranking second in the final leaderboard of the 4th RePSS Challenge. Compared to the winning team, HFUT-VUT, which achieved the RMSE of 11.89505 bpm, our result is only approximately 0.42 bpm higher, demonstrating the strong performance and competitiveness of our proposed method.
Table 2 compares our method with PhysFormer under diferent input settings. PhysFormer achieved the RMSE of 12.95383 bpm with RGB input, and a slightly improved 12.92008 bpm when using concatenated RGB and NIR frames. In contrast, the proposed method, with a more efective RGB-NIR fusion strategy, achieved a significantly lower RMSE of 12.31846 bpm, reflecting a relative improvement of about 4.9% over the RGB-only PhysFormer. This demonstrates the importance of both modality integration and efective fusion design to fully leverage the complementary information from RGB and NIR inputs.
This paper presents a multimodal HR estimation method that fuses RGB and NIR video information using a TDT architecture. Modality-specific spatio-temporal encoders are first applied to extract robust spatio-temporal features, providing a strong foundation for subsequent fusion. The TDT enables the model to capture subtle physiological variations in facial regions while efectively fusing RGB and NIR features. To further improve prediction accuracy, the model is trained with a composite loss function combining time-domain and frequency-domain components, guiding it to focus on both dominant frequency localization and waveform shape consistency. In the 4th RePSS Challenge, the proposed method achieved second place on the oficial test set, demonstrating its efectiveness and competitiveness in multimodal HR estimation. We believe that future improvements could lead to even better outcomes. (10)
This work is supported by the National Natural Science Foundation of China (62350068).
During the preparation of this work, the authors used ChatGPT in order to: Grammar and spelling check, Paraphrase and reword. After using this tool/service, the authors reviewed and edited the content as needed and took full responsibility for the publication’s content. [16] X. Niu, S. Shan, H. Han, X. Chen, Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation, IEEE Transactions on Image Processing 29 (2019) 2409–2423. [17] X. Li, I. Alikhani, J. Shi, T. Seppanen, J. Junttila, K. Majamaa-Voltti, M. Tulppo, G. Zhao, The obf database: A large face video database for remote physiological signal measurement and atrial ifbrillation detection, in: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE, 2018, pp. 242–249.