Reading Notes

Personal reading notes. Original paper: arXiv:2606.08063v1.

arxiv_id: 2606.08063v1

Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding? Reading Note

1. Paper Overview

This paper addresses the problem of significant performance degradation in Multimodal Large Language Models (MLLMs) under real-world visual interference. It proposes a new framework named Robust-U1. The authors argue that existing robustness enhancement methods have limitations: black-box feature alignment lacks interpretability, while white-box text reasoning cannot recover lost pixel-level details. To this end, the paper explores a core research question: Can MLLMs self-recover corrupted visual content to achieve robust understanding?

Robust-U1 endows MLLMs with explicit visual self-recovery capabilities. By combining supervised fine-tuning, dual-reward reinforcement learning, and multimodal reasoning in three stages, it achieves state-of-the-art (SOTA) robustness on real-world interference benchmarks and under adversarial interference.

2. Background and Motivation

Although MLLMs have achieved great success in visual understanding, their performance drops sharply when facing real-world visual interference such as system noise, compression artifacts, and adverse weather. The paper points out that existing robustness enhancement methods are mainly divided into two categories, both of which have deficiencies:

1. Implicit Adaptation: Aligns features of corrupted and clear images within the visual encoder through adversarial alignment. While this method can improve performance, it lacks interpretability and fails to explicitly model the interference process.
2. Text-based Reasoning: Uses explicit text reasoning chains to describe the type of interference and its semantic impact. This method enhances interpretability but is limited by the text modality and cannot represent pixel-level details for faithful visual understanding.

To address the above limitations, the authors propose a new paradigm where the model directly recovers corrupted images and performs multimodal reasoning. The paper compares these three robustness enhancement paradigms in the figure below:

As shown in Figure 1, (A) represents black-box feature alignment, (B) represents text-based reasoning, and (C) represents the Robust-U1 (Self-Recovery) proposed in this paper. It directly performs visual self-recovery and combines corrupted and recovered images for multimodal reasoning.

3. Core Method: Robust-U1 Framework

The Robust-U1 framework aims to endow MLLMs with explicit visual self-recovery capabilities. Although the original paper contains a framework overview figure (Figure 2), the corresponding image is not provided here. The following is a textual summary based on the paper description:

The framework includes three core training stages:

1. Stage 1: Supervised Fine-Tuning (SFT) for Visual Self-Recovery. Fine-tune a unified multimodal large language model using a large-scale real-world image reconstruction dataset to equip it with the basic capability to recover clear images from corrupted inputs, using Rectified-Flow Loss.
2. Stage 2: Enhancing Visual Quality via Reinforcement Learning. Adopt reinforcement learning with a dual-reward mechanism to further optimize the quality of recovered images.
3. Stage 3: Multimodal Reasoning for Robust Visual Understanding. Train the model to answer questions by jointly analyzing corrupted images and recovered images, thereby synthesizing more robust visual understanding.

3.1 Dual-Reward Reinforcement Learning Mechanism

In the second stage, to align with high visual quality, the authors design a reinforcement learning strategy containing dual rewards. This mechanism focuses on both pixel-level structural integrity and semantic-level consistency.

As shown in Figure 3, the dual-reward mechanism includes:

• Pixel-Level Structural Reward: Based on the SSIM (Structural Similarity) metric, ensuring the recovered image remains structurally consistent with the original clear image.
• Semantic Consistency Reward: Based on the CLIP model, ensuring the recovered image matches the original content at the semantic level.

Through this Flow-GRPO method, the model can balance detail restoration and semantic accuracy during the recovery process.

3.2 Multimodal Reasoning

In the third stage, the model receives not only the corrupted image but also the recovered image as input. The model is trained to use these two visual inputs for reasoning, thereby obtaining a more robust basis when answering visual questions.

4. Experimental Results and Analysis

The authors verified the effectiveness of Robust-U1 on multiple benchmarks, including the real-world interference benchmark R-Bench and general VQA benchmarks (such as MMMB, MMStar, RealWorldQA).

4.1 Visual Recovery Quality Comparison

The paper presents visual comparisons of recovered images under different training stages to prove the effectiveness of the training process.

As shown in Figure 4, as the training stages progress (from SFT to RL to final reasoning), the quality of recovered images improves significantly, with clearer details. This proves the enhancing effect of the staged training strategy on visual recovery capabilities.

4.2 Robustness Verification

To verify the direct contribution of recovered images to reasoning performance, the paper conducted visual verification analysis.

As shown in Figure 5, by comparing the recovery effects of different methods (green represents the proposed method), Robust-U1 performs excellently in maintaining image clarity and semantic accuracy. The analysis confirms that high-quality visual recovery directly enhances reasoning performance, establishing self-recovery as a key mechanism for achieving robust visual understanding.

Experimental results show that Robust-U1 significantly outperforms existing robust MLLMs on R-Bench. Furthermore, when facing adversarial interference, it maintains excellent performance on general VQA benchmarks with minimal performance degradation.

5. Summary

The Robust-U1 framework proposed in this paper is the first to endow MLLMs with explicit visual self-recovery capabilities, breaking through the limitations of relying solely on feature alignment or text reasoning in the past. Through a three-stage design of supervised fine-tuning, dual-reward reinforcement learning, and multimodal reasoning, the framework achieves pixel-level visual content reconstruction, thereby significantly improving the model’s robustness under real-world interference and adversarial attacks. The experimental analysis in the paper further confirms that high-quality visual recovery directly promotes the improvement of reasoning performance, providing a new direction for MLLM robustness research.