Reading Notes

Personal reading notes. Original paper: arXiv:2506.09718v2.

arxiv_id: 2506.09718v2

Paper Deep Dive: Non-Contact Health Monitoring During Daily Personal Care Routines

1. Paper Overview

This paper, titled “Non-Contact Health Monitoring During Daily Personal Care Routines,” is a collaborative effort by researchers from Qinghai University, Tsinghua University, the University of Washington, and other institutions. Addressing challenges such as environmental lighting changes, hand occlusion, and dynamic facial poses faced by remote photoplethysmography (rPPG) in long-term personal care scenarios (e.g., mirror routines in high-altitude environments), the paper proposes the LADH (Long-term Altitude Daily Health) long-term rPPG dataset and designs a multi-modal fusion model named FusionPhysNet.

The core contributions of the research are:

1. Constructing the first long-term rPPG dataset containing synchronized RGB and infrared (IR) videos, covering 5 common personal care scenarios.
2. Verifying that RGB and IR video fusion input significantly improves the accuracy and robustness of non-contact physiological monitoring (Heart Rate estimation MAE reaches 4.99 BPM).
3. Demonstrating that multi-task learning (simultaneously predicting Heart Rate, Respiration Rate, and SpO2) enhances model performance across multiple physiological metrics.

2. Research Background and Challenges

Traditional vital sign measurement methods (such as Heart Rate HR, Oxygen Saturation SpO2, and Respiration Rate RR) typically require bulky or invasive devices, limiting their practicality for continuous monitoring in daily life. rPPG technology utilizes cameras to detect minute changes in skin reflectance, offering a non-invasive alternative.

However, in real-world environments, rPPG faces numerous challenges:

• Environmental Interference: Clothing, hair, cosmetics, head movements, and changes in environmental lighting.
• Scene Limitations: Existing public benchmark datasets (e.g., PURE, UBFC-rPPG, SUMS) are mostly collected under controlled laboratory conditions, typically involving static sitting postures, and fail to capture physiological changes during daily personal care activities (such as brushing teeth or combing hair).
• Occlusion Issues: Frequent hand movements during personal care cause facial occlusion, affecting signal extraction.

To address these issues, this study focuses on daily personal care scenarios, aiming to fill the robustness research gap in long-term, comprehensive user studies.

3. LADH Dataset Details

The LADH dataset is the core foundation of this study, designed to support the evaluation of rPPG methods in daily personal care scenarios.

3.1 Data Collection Setup

Data collection was conducted in simulated daily personal care scenarios. The study recruited 21 graduate students (aged 23-28) as participants. The hardware configuration included:

• Camera: WN-12207K3321SM290 camera module, simultaneously capturing RGB and IR modal facial videos (640×400 resolution, 30 FPS).
• Physiological Ground Truth Devices: CMS50E Pulse Oximeter (for PPG and SpO2, sampling rate 20 Hz) and HKH-11C Respiration Sensor (for respiration patterns, sampling rate 50 Hz).
• Synchronization: All video frames and physiological signals were collected with millisecond-level precision via a data acquisition platform based on PhysRecorder.

The figure below shows the experimental setup during data collection, specifically the scenario of participants brushing their teeth, reflecting occlusion challenges in real-world environments.

(Extracted from embedded image on Page 1 of PDF, corresponding to Fig. 1 in the text: Data collection experimental setup during brushing)

3.2 Collection Protocol and Scenarios

The data collection protocol includes five states. Some participants underwent a 10-day longitudinal study (11 people), while the rest were single-day records (10 people). The specific scenarios are as follows:

1. Sitting Still: Wearing sensors, facing the camera, maintaining minimal movement.
2. Sitting Personal Care: Performing brushing and combing activities (including facial occlusion).
3. Standing Still: Same sensor configuration as sitting still.
4. Standing Personal Care: Repeating brushing and combing actions while standing.
5. Post-Exercise: Performing moderate-intensity exercise (squats, high knees, etc.) to induce physiological changes, followed by sitting recording.

The figure below visually illustrates the daily data collection protocol, with participants performing different activities in different states.

(Extracted from embedded image on Page 2 of PDF, corresponding to Fig. 2 in the text: Visual illustration of daily data collection protocol)

3.3 Dataset Comparison

Compared to existing datasets, LADH offers significant advantages. The table below summarizes the comparison between LADH and datasets such as PURE, UBFC-rPPG, MMPD, and SUMS:

LADH introduces challenges such as facial occlusion and varying indoor lighting, providing valuable resources for studying the robustness of non-contact physiological signal measurement.

4. Method: FusionPhysNet

To effectively utilize the LADH dataset and address multi-modal fusion problems, the authors proposed the FusionPhysNet model. This model is an extension based on the PhysNet architecture.

4.1 Input Embedding and Multi-modal Fusion

FusionPhysNet introduces an input embedding strategy to handle facial videos from two modalities (RGB and IR). The two video streams are preprocessed into frame sequences and undergo spatiotemporal feature extraction through a shared 3D convolutional encoder. By integrating features from both modalities, the model can more comprehensively understand an individual’s physiological state, effectively combining surface and deep physiological signals. This multi-modal fusion framework significantly enhances model performance in complex real-world scenarios.

4.2 Neural Network Architecture

The overall architecture of the model is shown in the figure. Based on the PhysNet backbone network, the authors introduced a modality-aware fusion mechanism in FusionNet. Specifically, a gated feature selection strategy is adopted to adaptively adjust the contribution of each modality based on global context representation. This design allows the model to dynamically emphasize the more informative modality under changing environmental conditions (such as lighting changes), thereby enhancing the robustness and generalization capability of the physiological signal estimation framework.

(Extracted from embedded image on Page 2 of PDF, corresponding to Fig. 3 in the text: FusionPhys model architecture, input is facial RGB and facial IR frames, PPG, RR, and SpO2 estimation tasks trained simultaneously)

4.3 Joint Training and Loss Function

Inspired by the multi-task training framework, the authors incorporated Respiration Rate (RR) estimation into the joint training mechanism, allowing simultaneous optimization of Heart Rate (HR), SpO2, and Respiration Rate (RR) prediction tasks. Additionally, a specifically designed loss function was introduced to jointly handle these three objectives. By optimizing these three targets simultaneously, the model is guided to learn more comprehensive and informative feature representations. The loss function formula is as follows:

$\text{Loss} = \text{MSE}_{\text{BVP}} + \text{MSE}_{\text{RR}} + 0.002 \times \text{MSE}_{\text{SpO2}} \times (100 - \text{Mean}(\text{SpO2}))$

Where $\text{MSE}_{\text{BVP}}$ represents the Mean Squared Error of the Blood Volume Pulse (BVP) signal, and $\text{MSE}_{\text{RR}}$ represents the Respiration Rate error.

5. Experimental Results and Analysis

5.1 Evaluation Settings

Evaluation employed both subject-independent and day-wise partitioning methods. Experiments found that day-wise partitioning enhances measurement accuracy, indicating that the model possesses a certain degree of generalization capability across data distribution changes on different days.

5.2 Key Findings

1. Multi-modal Fusion Advantage: Combining RGB and IR video inputs significantly improves the accuracy of non-contact physiological monitoring, even with frequent facial occlusion during brushing and combing.
2. Heart Rate Estimation Accuracy: On the LADH dataset, the model achieved a Mean Absolute Error (MAE) of 4.99 BPM for heart rate estimation.
3. Multi-task Learning Gain: Multi-task learning enhanced performance when simultaneously estimating heart rate and respiration rate, proving that joint optimization helps capture complex physiological dynamics.

6. Summary and Outlook

By constructing the LADH dataset and proposing the FusionPhysNet model, this paper effectively advances the application of non-contact health monitoring in daily personal care scenarios. The research not only addresses practical challenges such as lighting changes, occlusion, and dynamic poses but also validates the effectiveness of multi-modal (RGB+IR) and multi-task learning.

Future work may include:

• Expanding the dataset scale to cover more diverse populations and environments.
• Exploring more efficient lightweight models for deployment on edge devices (such as smart mirrors).
• Further investigating drift issues in long-term monitoring to ensure stability during long-term use.

Code and dataset are open-sourced at: https://github.com/McJackTang/FusionVitals.