Reading Notes

Personal reading notes. Original paper: arXiv:2606.14277v1.

arXiv Paper Reading Note: One Layer’s Trash is Another Layer’s Treasure

Paper Basic Information

• Paper Title: One Layer’s Trash is Another Layer’s Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs
• Paper ID: 2606.14277v1
• Authors: Yongru Chen, Kai Zhang, Zeliang Zong, Yuchen Lu, Wenming Tan, Ye Ren, Jilin Hu
• Affiliations: Hikvision Research Institute, Peking University, East China Normal University
• Core Topic: Visual Token Compression and Inference Acceleration in Large Vision-Language Models (LVLMs)

Research Background and Motivation

Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks but face substantial computational burdens in practical deployment. The paper points out that this burden primarily stems from the lengthy visual tokens generated when processing images. For instance, LLaVA-NeXT may generate up to 2,880 tokens when processing a 672 × 672 image, whereas text input typically consists of fewer than 100 tokens. This imbalance causes the computational load to be dominated by the visual modality, limiting the deployment of models in resource-constrained environments.

Existing visual token pruning methods (such as FastV) reduce computation by estimating token importance but suffer from a fundamental flaw: once a token is pruned at a specific layer, it becomes inaccessible to all subsequent layers. This irreversible information loss may cause the model to lose critical visual information in deeper networks, thereby compromising generation quality.

Core Finding: Layer-wise Attention Heterogeneity

To explain the aforementioned issue, the authors conducted empirical research to observe the visual region focus points at different layers. The paper argues that different network layers exhibit different visual region focuses, meaning the optimal token subset varies across layers.

The authors performed a visualization analysis using the FastV algorithm as an example to demonstrate the differences in focus on the same image across different layers. As shown in Figure 1, when processing an image of a bridge:

• Layer 2 primarily focuses on the bridge structure below.
• Layer 10 captures the sign with the bridge name.
• Layer 20 concentrates on the tower structure.

Figure 1 clearly demonstrates the hierarchical visual token attention patterns in the FastV algorithm. Colored areas represent important tokens, while gray areas represent unimportant tokens. This cross-layer heterogeneity indicates that static pruning at a single layer might discard instruction-related regions crucial for subsequent layers, leading to prediction errors. Only by integrating important visual information from multiple layers can the model generate accurate responses.

Method Details: ALVTS Framework

Based on the above observations, the authors propose the Adaptive Layer-wise Visual Token Selection (ALVTS) framework. This method breaks the traditional static token pruning paradigm, allowing each layer to perform dynamic token selection independently.

The core mechanism of ALVTS includes the following components:

1. Lightweight Token Selector: Before visual tokens enter the decoder layer, they are processed by a selector. The selector calculates an importance score for each token.
2. Dynamic Routing and Skip:
   • High-score (important) tokens participate in the computation of the current layer.
   • Low-score (unimportant) tokens directly skip the layer, avoiding unnecessary operations.
3. Seamless Reintegration: After the decoder layer output, the processed tokens are seamlessly merged with the skipped tokens, restoring the complete token sequence for use in the next layer.

Figure 2 illustrates the overall architecture of ALVTS. In each LLM decoder layer, the lightweight token selector uses a low-rank approximation of the full attention mechanism to calculate importance scores. Important tokens participate in computation, while secondary tokens skip the layer. The two paths of tokens are reintegrated before the next layer, achieving adaptive compression across the model.

The paper notes that this token selection module is based on importance consistency constrained low-rank approximation. This design enables the selector to closely simulate the token selection behavior of the full attention mechanism, effectively capturing its fundamental patterns without requiring model retraining, facilitating practical deployment.

Experimental Results and Comparison

The authors conducted extensive experiments on mainstream LVLM architectures such as LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL to verify the effectiveness of ALVTS.

Quantitative Results

Experimental results indicate that ALVTS achieves a good balance between efficiency and accuracy:

• Compression Rate: Achieved 89% visual token compression.
• Accuracy Retention: Retained 96.7% of the original model’s performance.
• Inference Speed: Achieved a 1.6x inference speedup.

Qualitative Comparison

To intuitively demonstrate the effects of different methods, the paper compares the pruning situations of FastV and ALVTS. As shown in Figure 5, gray masks represent pruned (skipped) regions.

Figure 5 presents a case comparison between FastV and ALVTS. Alongside the original image, its pruned versions are displayed. It can be seen that ALVTS can more flexibly retain key visual information required by different layers, whereas FastV, due to its static pruning strategy, may lose details needed in subsequent layers at certain layers.

Summary and Contributions

The main contributions of the paper are summarized as follows:

1. Revealed Existing Limitations: Pointed out the fundamental flaw of existing token pruning strategies, where pruning at a specific layer makes tokens inaccessible to subsequent layers, causing irreversible information loss.
2. Proposed ALVTS Framework: Introduced adaptive layer-wise visual token selection instead of using a globally fixed token subset. This method eliminates redundant computation while retaining necessary visual semantics throughout the model.
3. Efficient Selector Design: Developed an efficient low-rank token selector that maintains high fidelity with the full attention mechanism without requiring model retraining.
4. Comprehensive Evaluation: Comprehensive evaluation across multiple LVLMs and benchmarks proves that ALVTS achieves a superior efficiency-accuracy trade-off.

Through the core insight that “one layer’s trash is another layer’s treasure,” ALVTS provides a new solution for efficient inference in LVLMs, effectively alleviating the computational bottleneck caused by visual tokens.