Reading Notes

Personal reading notes. Original paper: arXiv:2303.07944v1.

arxiv_id: 2303.07944v1

Paper Deep Dive: Non-Contrastive Unsupervised Learning of Physiological Signals from Video

1. Paper Basic Information

• Paper Title: Non-Contrastive Unsupervised Learning of Physiological Signals from Video
• Paper ID: 2303.07944v1
• Authors: Jeremy Speth, Nathan Vance, Patrick Flynn, Adam Czajka
• Affiliation: University of Notre Dame
• Core Contribution: Proposed the first non-contrastive unsupervised learning framework (SiNC) for physiological signal regression, eliminating reliance on contact PPG sensor labels.

2. Research Background and Motivation

Camera-based vitals estimation is a significant direction in the field of remote health monitoring. By extracting weak periodic signals (such as blood volume pulses and respiration) from RGB video, low-cost remote health monitoring can be achieved. Current remote photoplethysmography (rPPG) research is primarily driven by deep learning, but existing methods mostly rely on supervised training using benchmark datasets with ground truth from contact PPG sensors.

However, collecting large-scale, diverse labeled video data faces significant challenges:

1. Large Data Volume: High-quality video data occupies a large amount of storage.
2. Collection Difficulty: It is difficult to record diverse populations representing real-world activities in a laboratory environment.
3. Synchronization and Noise: Synchronizing contact measurements with video is technically challenging, and contact measurements themselves contain noise.

Although Contrastive Unsupervised Learning provides a new perspective for solving data scarcity, existing unsupervised rPPG methods are all based on contrastive learning frameworks, requiring the construction of positive and negative sample pairs. This paper proposes a Non-Contrastive Unsupervised Learning framework, which can directly learn physiological signal features from unlabeled videos using only the video data itself by imposing periodic priors and frequency domain constraints.

3. Core Method: SiNC Framework

The method proposed in this paper is named SiNC (Signal from Non-contrastive). Its core idea is to use strong prior knowledge of physiological signals (such as bandwidth limitations and periodicity) to constrain the solution space without calculating distances between samples or contrastive losses.

3.1 Overall Architecture

As shown in Figure 1, the SiNC framework contrasts sharply with traditional supervised learning and contrastive unsupervised learning.

• Supervised Learning: Requires ground truth labels $y$, trained by minimizing the error between the predicted waveform and the ground truth.
• Contrastive Unsupervised Learning: Requires constructing positive and negative sample pairs, learning features by pulling similar samples closer and pushing dissimilar samples apart.
• SiNC (This Paper’s Method): Directly imposes loss functions on the frequency domain of the predicted waveform. By penalizing power outside the physiological bandwidth, encouraging sparse distribution within the bandwidth, and maximizing variance between batches, the model can automatically learn visual features related to physiological signals without any labels or sample pair comparisons.

3.2 Loss Function Design

To guide the model to learn in the absence of labels, the authors designed three key loss functions, all calculated in the frequency domain (after FFT transformation):

1. Bandwidth Loss ($L_B$): Physiological signals (such as heart rate) usually lie within a specific frequency range. This loss penalizes the power of the model’s predicted signal outside the normal physiological bandwidth. This forces the model to ignore noise frequencies unrelated to physiological signals and learn invariance to irrelevant frequencies.

2. Sparsity Loss ($L_S$): Real physiological signals typically manifest as sparse peaks in the frequency domain (i.e., energy concentrated near specific frequencies). This loss encourages a sparser power spectrum distribution within the physiological bandwidth, meaning energy is concentrated near the main peak rather than being uniformly distributed.

3. Batch-wise Variance Loss ($L_V$): To prevent model collapse (i.e., all inputs predicting the same constant signal), this loss encourages sufficient variance between the power spectra corresponding to different batch inputs. This ensures the model can capture the differences in physiological signals across different video samples.

The total loss function is the weighted sum of these three: $L_{total} = L_B + L_S + L_V$.

4. Experimental Verification and Results

4.1 Training on Unlabeled Data

This paper conducted the first experiment using non-rPPG-specific, unlabeled video data to train a robust heart rate estimator. This means the training data does not require specifically collected physiological signals; ordinary video data is sufficient. This greatly lowers the data collection threshold, allowing the model to be trained using more extensive and representative video data.

4.2 Loss Function Ablation Study

To verify the effectiveness of each loss component, the authors conducted ablation studies on the UBFC-rPPG dataset. Figure 2 shows the model prediction results using different loss combinations after training for 20 epochs.

• The first column shows the situation using only partial losses.
• Subsequent columns show the effect after combining all losses.
• Experimental results indicate that using certain losses alone may lead to undesirable spectral distributions, while combining bandwidth, sparsity, and variance losses (i.e., the complete SiNC framework) can most effectively extract clear periodic signal features.

4.3 Performance Evaluation and Visualization

Figure 3 further demonstrates the model’s performance or signal quality comparison in different scenarios. Through visualization analysis, it can be seen that the SiNC method can recover waveforms highly consistent with real physiological signals without ground truth labels. This proves that deep learning models indeed possess the ability to “discover” physiological signals from video solely based on periodic priors and frequency domain constraints.

5. Summary and Outlook

The SiNC framework proposed in this paper is an important breakthrough in the field of camera-based vital sign measurement.

1. Eliminate Label Dependency: It achieves non-contrastive unsupervised learning for physiological signal regression for the first time, breaking the dependence of rPPG research on contact sensor labels.
2. Data Scalability: It makes it possible to train models using massive amounts of unlabeled video data, helping to improve the robustness of the model under different skin tones, lighting, and motion conditions.
3. Generalization Potential: Due to limited inductive biases and reliance only on periodicity assumptions, this method can theoretically be extended to extract other periodic or quasi-periodic signals from video (such as respiration rate), providing a new technical path for multi-parameter contactless health monitoring.

The code has been made public with the paper, providing a reproducible foundation for subsequent research. This work marks a key step for video-based physiological signal analysis moving from “supervised learning” to “self-supervised/unsupervised learning”.