Reading Notes

Personal reading notes. Original paper: arXiv:2111.09748v1.

arxiv_id: 2111.09748v1

Paper Deep Dive: The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video

1. Paper Overview

This paper proposes a novel method for estimating Remote Photoplethysmography (rPPG). rPPG technology aims to measure blood volume changes by observing videos of human faces or skin, thereby inferring physiological signals such as heart rate. Unlike current rPPG methods that require expensive physiological ground truth data for supervised learning, this paper adopts a fully self-supervised training method that requires no labeled data.

The core innovation lies in utilizing Contrastive Learning, combined with weak prior assumptions regarding the target signal’s frequency and temporal smoothness. The authors evaluated this method on four rPPG datasets, and the results show that its performance is comparable to, or even better than, recent supervised deep learning methods, without requiring any labels. Additionally, the paper introduces a Learned Saliency Resampling Module, which not only reduces the need for handcrafted features but also provides interpretability for model behavior.

The figure below shows the overall input-output flow of the model: from face video input alone, the model can infer the PPG signal and heart rate, and generate interpretable saliency heatmaps showing the facial regions the model focuses on (e.g., the center of the forehead).

2. Research Background and Pain Points

Understanding human physiological states is crucial in fields such as health monitoring, human resource management, and human-computer interaction. Traditional physiological signal estimation methods (such as ECG or contact PPG) require physical contact with the subject, involve cumbersome setup, and limit scalability.

In recent years, the development of computer vision and machine learning has promoted research on passively monitoring physiological information from videos. Existing rPPG methods generally fall into three stages: preprocessing (reducing interference), PPG signal extraction, and heart rate estimation. Early methods relied on signal processing and handcrafted features (such as color or motion in specific regions), while recent research tends to use deep learning to learn feature representations that are robust to lighting changes, motion interference, etc.

However, supervised deep learning faces a major bottleneck in the rPPG field: the cost of labeled data is extremely high. To train models, subjects must wear contact PPG or ECG sensors while videos are being collected. This is not only difficult to scale but also means that changes in data distribution (e.g., from training to deployment) can severely affect model performance in real-world scenarios.

3. Core Method: Self-Supervised Framework Based on Contrastive Learning

To address the issue of expensive data labeling, this paper proposes a perspective based on self-supervised learning. The authors constructed a contrastive learning framework based on three basic assumptions about the target physiological signal:

1. A1 Frequency Range Assumption: The target signal lies within a specific range (set to 40 to 250 beats per minute), covering the vast majority of human heart rates.
2. A2 Temporal Smoothness Assumption: The target signal usually does not change drastically within short time intervals (i.e., the heart rate at time $t$ is similar to that at time $t+W$, where $W$ is on the order of seconds).
3. A3 Visual Dominance Assumption: The signal has some visible manifestation within the target frequency range and is the dominant visual signal within that range.

Based on these assumptions, the model can be trained without ground truth labels. The figure below shows an overview of the overall architecture of this method. The process first samples a video clip of length $W$ from the source video, followed by optimization through a contrastive learning loss function, aiming to extract feature representations related to physiological signals.

The advantage of this method is that it does not require expensive ground truth physiological training data but instead uses the statistical properties of the video data itself (such as frequency distribution and temporal continuity) to guide network learning. By setting a contrastive learning loss, the model can distinguish between clips containing the target physiological signal and those that do not, thereby learning to regress the PPG signal and track the heart rate.

4. Interpretability Enhancement: Saliency Resampling Module

Deep learning models are often regarded as “black boxes,” and it is difficult to understand how the model makes predictions, especially without labeled data. To address this issue, the authors introduced a front-end Saliency-based Sampling Module.

Inspired by related work, this module aims to emphasize the parts of the input data most relevant to the backend PPG estimator. By allowing the model to learn where to sample the input image, this method not only reduces the need for manually designed features but also provides transparency regarding model behavior.

4.1 Impact of Regularization on Saliency

The authors investigated the impact of regularization on the computed saliency estimates. By adjusting regularization parameters, the smoothness and focus of the saliency map can be controlled, helping the model more accurately locate the regions in the face with the strongest physiological signals.

4.2 Examples of Interpretable Behavior

The saliency map is not just a byproduct; it can reveal the spatial location of the signal of interest discovered by the model. As shown in Figure 4, the saliency map generated by the model is mainly concentrated on the subject’s forehead, nose, and cheeks. This aligns with the traditional understanding of regions where rPPG signals are strongest, proving that the model has learned reasonable spatial features rather than random noise.

This interpretability is crucial for understanding model failure modes. For example, if the saliency map is concentrated on non-skin areas, it may indicate that the model is interfered with by lighting or motion artifacts. This module can be applied independently to both supervised models and contrastive models, demonstrating good generality.

5. Experimental Evaluation and Results

The authors evaluated the proposed method on four rPPG datasets. Experimental results show that although no labeled data was used for training, the performance achieved by this self-supervised method is comparable to recent supervised deep learning methods, and even superior in some cases.

Key experimental findings include:

• Effectiveness of Unlabeled Training: Demonstrates the feasibility of a self-supervised framework based on contrastive learning and weak prior assumptions for rPPG tasks.
• Robustness: The proposed loss function is robust to synchronization issues in the ground truth.
• Module Generality: The saliency sampling module can be attached to both supervised baselines and contrastive models, bringing performance improvements or enhanced interpretability in both cases.

Additionally, the authors unified existing free PPG video datasets and released a complete training and evaluation code pipeline to promote reproducible research in this direction.

6. Summary and Contributions

The main contributions of this paper can be summarized as follows:

1. Self-Supervised rPPG Framework: First demonstrated that deep neural networks can be trained on unlabeled video data to estimate PPG signals and heart rate using only contrastive learning and weak prior assumptions.
2. Novel Loss Function: Proposed supervised and unsupervised loss functions that are robust to ground truth synchronization issues.
3. Interpretability Module: Introduced a learnable saliency resampling module that provides spatial transparency for model behavior and reduces reliance on handcrafted features.
4. Open Source and Reproducibility: Unified datasets and released complete code, encouraging reproducible progress in the emerging direction of computer vision health monitoring.

This work provides a new technical path for low-cost, non-contact health monitoring. By eliminating the reliance on expensive labeled data, it greatly expands the potential application scenarios of rPPG technology.