Reading Notes

Personal reading notes. Original paper: arXiv:2404.06483v2.

arxiv_id: 2404.06483v2

RhythmMamba: Fast and Lightweight Remote Physiological Measurement Based on State Space Models

1. Research Background and Core Challenges

Remote Photoplethysmography (rPPG) is a technology for non-contact measurement of physiological signals through facial video, holding significant potential in healthcare, affective computing, and anti-counterfeiting. Its core principle involves using optical means to measure changes in blood volume within subcutaneous vessels due to cardiac contraction and relaxation. These changes result in periodic color signal variations on the imaging sensor that are invisible to the human eye.

Despite the broad application prospects of rPPG, existing deep learning methods face a trade-off between two core challenges:

1. Understanding Periodic Patterns in Long Contexts: rPPG signals exhibit significant periodicity, requiring models to capture long-range temporal dependencies.
2. Handling Spatiotemporal Redundancy in Video Clips: Video data contains substantial spatiotemporal redundancy, necessitating efficient computational complexity.

Traditional Convolutional Neural Networks (CNNs) have limited receptive fields and struggle to capture long-range dependencies. While Transformer-based architectures can capture long dependencies, their quadratic computational complexity is inefficient when processing long video sequences. This trade-off between computational complexity and the ability to capture long-range dependencies hinders the deployment of rPPG solutions on mobile devices.

Recently, Mamba models have provided new insights for rPPG tasks by achieving long-range dependency modeling while maintaining linear complexity, thanks to their Selective State Space Model (SSM) and parallel scanning mechanisms. However, directly applying Mamba to rPPG tasks yields suboptimal results because embedding spatiotemporal information into token sequences causes spatial information to significantly interfere with Mamba’s understanding of temporal information.

2. RhythmMamba Overall Architecture

To address the above issues, the authors propose RhythmMamba, a remote physiological measurement architecture based on state space models. This model treats the rPPG task as a time series task and enhances Mamba’s learning capability for rPPG signals through specific module designs.

As shown in Figure 3, the overall framework of RhythmMamba mainly consists of four parts:

1. Frame Stem: Extracts rPPG features using diff-fusion, self-attention mechanisms, and frame average pooling, embedding all spatial information into the channels.
2. Multi-temporal Constraint Mamba: The core backbone network, which enhances learning of periodicity through a multi-temporal constraint mechanism.
3. Frequency Domain Feed-forward: Performs spatial interaction between channels in the frequency domain to better distinguish the periodicity of rPPG signals.
4. rPPG Predictor Head: Outputs the final physiological signal.

The input is an RGB video $X \in \mathbb{R}^{3 \times T \times H \times W}$, which becomes $X_{stem} \in \mathbb{R}^{T \times C}$ after processing by the Frame Stem, where $C$ is the number of channels and $T$ is the sequence length. This design compresses the spatial dimension into the channel dimension, allowing the model to focus on time series modeling.

3. Key Technical Innovations

The innovations of RhythmMamba are mainly reflected in the deep adaptation to rPPG signal characteristics, specifically including the following three aspects:

3.1 Frame Stem and Temporal Perspective

The authors found that directly embedding spatiotemporal information into token sequences interferes with Mamba’s understanding of time series. Therefore, a Frame Stem was designed to embed the spatial information of a single frame into the channels. This design converts the rPPG task into a pure time series task, allowing the periodic changes of the pulse wave to be modeled as state transitions within the state space.

3.2 rPPG Modeling from State Transition Perspective

Considering the linear modeling characteristics of Mamba, the phase changes of rPPG signals can be regarded as state transitions within the state space. The periodic nature of rPPG allows signals to be represented by a finite number of states. As shown in Figure 1, $h(t)$ represents the state vector, $x(t)$ represents the input vector, and $y(t)$ represents the output vector. Through this perspective, the model can more effectively capture the periodic fluctuations of the pulse wave.

3.3 Multi-temporal Constraint and Frequency Domain Feed-forward

To further enhance Mamba’s learning capability for rPPG signals, the authors designed two key modules:

• Multi-temporal Constraint: By allowing a single Mamba block to learn the same sequence with different time lengths, the model can be constrained by both the periodicity of long sequences and the trends of short sequences simultaneously.
• Frequency Domain Feed-forward: Temporal features learned by Mamba undergo spatial interaction between channels in the frequency domain. This frequency domain operation helps better distinguish the periodic essence of rPPG signals, compensating for the lack of spatial feature interaction in pure temporal modeling.

4. Experimental Results and Performance Evaluation

The authors conducted extensive experiments on multiple datasets, including intra-dataset and cross-dataset scenarios. Experimental results show that RhythmMamba significantly improves computational efficiency while maintaining high accuracy.

As shown in Figure 2, in the intra-dataset test on the MMPD dataset, RhythmMamba demonstrated an excellent balance between performance and efficiency:

• Throughput: Increased by 319% compared to existing methods.
• Peak GPU Memory: Only 23%.
• Accuracy: Reached State-of-the-Art (SOTA) levels.

The diameter of the circles in the figure represents the peak GPU memory. It can be seen that RhythmMamba is not only faster and lighter but also, as the scale increases, these advantages become more pronounced due to its linear complexity. This proves that the model is highly suitable for deployment on mobile devices.

5. Summary and Outlook

RhythmMamba is the first work to apply State Space Models (SSM) to the field of rPPG. Its main contributions include:

1. Proposed RhythmMamba based on SSM, modeling periodic changes as state transitions, and combining multi-temporal constraints and frequency domain feed-forward to learn quasi-periodic patterns of rPPG.
2. Designed the Frame Stem to embed spatial information into channels, reducing the dimension of state transitions and solving the problem of spatial information interfering with temporal understanding.
3. Achieved SOTA performance in multiple scenarios while possessing extremely high inference speed and extremely low memory consumption.

This research provides an efficient, lightweight, and accurate solution for remote physiological measurement, laying a solid foundation for future health monitoring applications in mobile devices and edge computing scenarios. The code has been open-sourced for further exploration and verification by the community.