Reading Notes

Personal reading notes. Original paper: arXiv:2606.12809v1.

arxiv_id: 2606.12809v1

MLUBench: Deep Dive into the Benchmark for Lifelong Unlearning of Multimodal Large Models

1. Research Background and Problem Definition

As Multimodal Large Language Models (MLLMs) demonstrate excellent multimodal reasoning capabilities in various applications, these models are typically trained on massive amounts of multimodal data, raising concerns regarding data privacy and copyright. Therefore, Machine Unlearning—removing specific data from a trained model—becomes crucial.

The paper points out that in real-world scenarios, unlearning requests often do not arrive simultaneously but are submitted sequentially over time. This practical setting introduces the challenging problem of MLLM Lifelong Unlearning. In this scenario, the MLLM must continuously forget multimodal information while retaining its general capabilities.

As shown in Figure 1, this process requires the model to forget information in specified knowledge sets while retaining other information when facing a series of sequential unlearning tasks. However, existing unlearning methods often lead to cumulative performance degradation of the model when facing such continuous operations.

2. Introduction to MLUBench Benchmark

To fill the gap in scale and scope of existing evaluation benchmarks, the authors propose MLUBench, a large-scale, diverse benchmark specifically designed to evaluate MLLM lifelong unlearning.

2.1 Benchmark Composition

MLUBench includes the following core elements:

• Entity Scale: Covers 127 widely recognized real-world entities.
• Category Distribution: Spans 9 different categories to ensure data diversity.
• Data Volume: Contains 5,105 associated images and 15,414 Visual Question Answering (VQA) pairs.

This benchmark organizes these entities into a series of unlearning tasks, providing a comprehensive platform for evaluating the long-term performance of unlearning algorithms. Figure 2 shows the overall overview of MLUBench, clearly presenting the relationship between entity categories and data composition.

2.2 Data Construction

During the construction of the benchmark, the authors adopted specific prompt templates to extract entity information. Figure 3 shows the prompt template used for entity extraction, which helps ensure the accuracy and consistency of the data in the benchmark.

3. Core Findings and Challenges

Using MLUBench, the authors conducted a comprehensive evaluation of existing unlearning methods and revealed two key findings:

3.1 Severe Cumulative Performance Degradation

The paper argues that lifelong unlearning leads to severe cumulative performance degradation in both unlearning quality and model utility. For example, the unlearning quality of the GA method drops from 0.38 on the first task to 0.01 after subsequent unlearning operations. This indicates that existing methods struggle to cope with continuous unlearning requests.

3.2 Unique Challenges of Multimodal Alignment

The paper further points out that, unlike unimodal models, MLLM lifelong unlearning faces a fundamental constraint: multimodal alignment must be maintained. The authors demonstrate through experiments that even if unlearning operations are applied to a single modality, it can catastrophically disrupt this alignment, leading to model performance collapse. This is a unique challenge distinguishing MLLM lifelong unlearning from unimodal unlearning.

4. Proposed Method: LUMoE

To address the above challenges, the authors propose an effective method named LUMoE (Lifelong Unlearning with Mixture-of-Experts).

4.1 Methodology Principle

Inspired by Mixture-of-Experts (MoE) models, the core idea of LUMoE is to no longer continuously change the weights of the MLLM, but instead adopt switchable Low-Rank Adaptation (LoRA) adapters as “experts” for specific unlearning tasks.

4.2 Working Mechanism

The method includes a powerful Gate Module responsible for guiding inputs to appropriate adapters, thereby effectively handling unlearning requests. This design aims to alleviate the degradation problems faced by existing baseline methods.

5. Summary

The main contributions of this paper are summarized as follows:

1. Problem Definition: Investigated the practical and challenging problem of MLLM lifelong unlearning and pointed out that maintaining multimodal alignment is a unique challenge of this problem.
2. Benchmark Construction: Introduced MLUBench, a large-scale benchmark containing 127 entities and 9 categories, for evaluating MLLM lifelong unlearning.
3. Experiments and Method: Revealed the performance degradation issues of existing methods through experiments and designed the LUMoE method, establishing a strong performance standard for future research in this field.

The authors have open-sourced the source code and the MLUBench dataset to promote further research in this field.