Reading Notes

Personal reading notes. Original paper: arXiv:2404.13449v1.

arxiv_id: 2404.13449v1

Paper Deep Dive: SiNC+ - Adaptive Camera-based Vitals Monitoring based on Unsupervised Learning

1. Research Background and Motivation

With the development of computer vision technology, camera-based vitals monitoring has become an important direction for non-contact health monitoring. By extracting weak periodic signals from RGB videos, such as Blood Volume Pulse (BVP) and respiratory signals, low-cost remote physiological parameter estimation can be achieved. This core technology is commonly known as Remote Photoplethysmography (rPPG).

Although deep learning has made significant progress in the field of rPPG, existing methods mainly rely on supervised learning, requiring large amounts of video data with ground truth labels from contact sensors (such as PPG or ECG) for training. However, collecting such data faces significant challenges:

1. Large Data Volume and High Quality Requirements: Long-duration high-quality video recording is needed.
2. Insufficient Diversity: It is difficult to cover diverse skin tones, lighting conditions, and motion scenarios in a laboratory environment.
3. Synchronization Difficulties: Precise synchronization between video and contact physiological signals is technically challenging, and contact measurements themselves contain noise.

To address the data scarcity problem, Unsupervised Learning is considered a promising solution. However, existing unsupervised rPPG methods mostly adopt Contrastive Learning, which requires constructing positive and negative sample pairs and typically lacks direct utilization of prior information about periodic signals. Furthermore, contrastive learning usually requires multi-subject datasets to form negative sample pairs, limiting its application in single-subject personalized scenarios.

2. Core Method: SiNC+ Framework

This paper proposes SiNC+ (Signal-based Non-contrastive), which is the first non-contrastive unsupervised learning framework for signal regression. The core idea of this framework is: with only minimal assumptions about the periodicity and limited bandwidth of the signal, the Blood Volume Pulse signal can be directly mined from unlabeled videos.

2.1 Framework Overview

SiNC+ discards the traditional contrastive learning paradigm (i.e., no need to calculate distances between samples or construct positive-negative pairs) and instead directly imposes constraints on the predicted signal in the frequency domain. Its loss function consists of three main parts, aiming to shape the power spectrum characteristics of the predicted signal:

1. Bandwidth: Penalizes power outside the target physiological frequency band, forcing the model to focus on relevant frequencies.
2. Sparsity: Encourages power within the target band to concentrate near the peak frequency, simulating the spectral characteristics of real physiological signals.
3. Batch-wise Variance: Encourages variance in the power spectrum between different input samples, preventing the model from collapsing to a single solution.

The figure below shows a comparison of the SiNC+ framework with traditional supervised learning and contrastive unsupervised learning. It can be seen that SiNC+ directly applies the loss to the prediction results, learning features by shaping the frequency spectrum, rather than relying on labels or distance metrics between sample pairs.

Figure 1: SiNC+ Framework Overview (from embedded figure on page 2 of PDF, corresponding to Figure 1 description in the text). Shows the differences in loss calculation between supervised, contrastive unsupervised, and non-contrastive unsupervised (this paper’s method).

2.2 Loss Function Design

The loss function $L_F$ of SiNC+ can be expressed as: $L_F = L_{bandwidth} + L_{sparsity} + L_{variance}$

• $L_{bandwidth}$: By restricting the power spectrum to be zero outside a specific physiological bandwidth (e.g., heart rate 0.8-3.5 Hz), it learns invariance to irrelevant frequencies.
• $L_{sparsity}$: Encourages a sparse distribution of the power spectrum within the bandwidth, meaning energy is concentrated on the fundamental frequency and its harmonics.
• $L_{variance}$: Encourages variance of the power spectrum at the batch level, ensuring the model can distinguish physiological features of different samples and avoid outputting a constant signal.

This design allows the model to be trained without contact ground truth labels, relying solely on visual features in the video data and physical priors of physiological signals.

3. Technical Details and Principle Analysis

3.1 Why Choose Non-contrastive Learning?

Existing unsupervised rPPG methods mostly adopt contrastive learning (e.g., Gideon et al. [8]), creating negative samples through frequency resampling. However, this method has limitations:

• Imprecise Negative Samples: The resampled frequency is known; directly penalizing the difference from a known spectrum is more accurate than “repelling” pairs.
• Spectral Overlap: Even if heart rates differ, the zero-crossing intervals of the main lobes produced by Fast Fourier Transform (FFT) may be wide (e.g., 30 bpm), causing spectral overlap between samples with different heart rates, making the “repulsion” operation inaccurate.

SiNC+ avoids the complexity of constructing negative sample pairs by directly optimizing the spectral shape. The paper points out that even if different subjects have the same heart rate, the batch variance constraint ensures the model learns effective feature representations.

3.2 Spectral Characteristics Analysis

To illustrate the importance of spectral sparsity and bandwidth constraints, the paper analyzes the characteristics of FFT decomposition. The figure below shows the distribution of zero crossings of the spectral main lobes, explaining why simple spectral distance metrics may fail in contrastive learning (spectral overlap problem).

Figure 2: Spectral Main Lobe Zero Crossing Analysis (from embedded figure on page 2 of PDF, corresponding to Figure 2 description in the text). Shows the situation of spectral overlap under different heart rates, explaining why negative sample repulsion in contrastive learning may be inaccurate.

By encouraging a sparse power spectrum, SiNC+ can more accurately locate the fundamental frequency and effectively distinguish signals even when spectral overlap exists.

4. Experiments and Application Scenarios

4.1 Generality and Cross-domain Application

A significant advantage of SiNC+ is its generality. Since the framework relies only on bandwidth constraints and periodicity assumptions, the same method can be applied to different physiological signals by adjusting the target signal bandwidth constraints.

• Pulse Monitoring: Uses typical heart rate bandwidth (approx. 0.8-3.5 Hz).
• Respiration Monitoring: Successfully achieves unsupervised camera-based respiration estimation by adjusting the bandwidth constraint to the respiration frequency range (approx. 0.1-0.5 Hz). This is the first experiment to utilize unsupervised learning for remote respiration estimation.

4.2 Training on Non-rPPG Specific Data

Most existing rPPG research relies on benchmark datasets specifically collected for rPPG. This paper conducts the first experiment using unlabeled video data not specifically intended for rPPG to train a robust pulse rate estimator. This proves that SiNC+ can learn visual features from general video data, greatly expanding potential training data sources.

4.3 Personalization and Adaptive Fine-tuning

Traditional unsupervised deep learning methods are usually aimed at training on large-scale datasets. SiNC+ demonstrates its fine-tuning capability on single-subject data. By fine-tuning on short segments of unlabeled video from a single subject, the model can achieve personalized and adaptive signal regression. This is significant for privacy-sensitive application scenarios (such as home health monitoring), allowing the model to adapt to specific users’ physiological characteristics and imaging conditions on local data.

5. Summary and Outlook

The SiNC+ framework proposed in this paper brings a new paradigm to camera-based vitals monitoring. Its main contributions include:

1. Proposed the first non-contrastive unsupervised learning framework for signal regression.
2. Successfully applied SiNC to contactless pulse and respiration waveform measurement.
3. First demonstrated the feasibility of training remote pulse estimators on unlabeled, non-rPPG specific video data.
4. Verified the effectiveness of the framework for personalized fine-tuning on single-subject data.

SiNC+ effectively avoids reliance on expensive labeled data by utilizing frequency domain priors and batch variance constraints. In the future, as the framework is validated on more physiological signals and complex scenarios, it is expected to promote non-contact health monitoring technology from the laboratory to actual deployment, especially in the context of growing data privacy and personalization needs.