Reading Notes

Personal reading notes. Original paper: arXiv:2606.14210v1.

arxiv_id: 2606.14210v1

Paper Reading Note: From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models

1. Research Background and Motivation

With the increasing deployment of Large Language Models (LLMs) in privacy-sensitive domains (such as finance and healthcare), users face a dilemma: using external APIs carries the risk of data leakage, while local full-scale deployment incurs excessive computational costs. To address this, Split Learning has emerged as a paradigm for LLM fine-tuning and inference under resource-constrained conditions, referred to as Split-LLM.

In a typical Split-LLM architecture, the model is divided into three parts: the Head and Tail are retained by the Data Party who owns private data, while the Body is deployed on the Model Party who possesses computational resources. Although this design aims to isolate sensitive data, the paper points out that existing research primarily focuses on Input Prompt leakage caused by inversion attacks via intermediate representations, while the potential risk of sensitive information leakage in Generative Response Outputs has not been fully explored.

The authors argue that in high-risk domains, both input prompts and output responses may contain highly sensitive information (such as investment strategies or proprietary insights). Therefore, the paper aims to systematically investigate how information leaks jointly from both ends of the model (input and output) during the Split-LLM generation process, and proposes corresponding attack and defense methods.

2. Problem Definition and Threat Model

The paper defines the problem based on a typical “Head-Body-Tail” (HBT) Split-LLM scenario.

2.1 Architecture Setup

The Model Party controls a complete pre-trained LLM ($M_{full}$) and divides it into three segments:

1. Head ($\bar{M}_h$): Contains the embedding layer and initial several decoder/encoder layers.
2. Body ($\bar{M}_b$): Contains the majority of the intermediate layers, retained on the Model Party server.
3. Tail ($\bar{M}_t$): Contains the final several layers and the output projection layer.

The Data Party owns lightweight Head and Tail segments. During the inference phase, the system performs multiple forward passes via an autoregressive generation mechanism. The Model Party can obtain the output activations $H$ from the model Head and the output activations $T$ from the model Body.

2.2 Threat Model

The paper assumes the Model Party is an “honest-but-curious” attacker. The attacker possesses the following information during the inference phase:

• Original model segments and the fine-tuned Body model.
• Intermediate activations $H$ output by the model Head and activations $T$ output by the model Body.
• (Optional) A small amount of auxiliary data similar to the Data Party’s private data distribution.

Figure 1 Description: Figure 1 in the paper illustrates the threat model for dual-data reconstruction attacks. The figure depicts the interaction flow between the Data Party and the Model Party, including the Split-LLM fine-tuning phase and the inference phase. During the inference phase, the Model Party utilizes the obtained intermediate activations to attempt to reconstruct the Data Party’s private input $X$ and the generated response $Y$. The figure intuitively demonstrates how the attacker leverages leaked information from both ends of the model to perform data reconstruction.

3. Attack Method: PIDI

To reveal new vulnerabilities in Split-LLM, the authors propose a dual-sided attack method named Patched Model Inversion with Dual-Sided Initialization (PIDI). This is a two-stage attack aimed at simultaneously reconstructing private input prompts and output responses.

3.1 Phase 1: Dual-Sided Initialization (DSI)

In this phase, the authors leverage the autoregressive nature of text generation to construct initial estimates of private inputs and responses.

• Input Branch: Combines model completion and SIP (Split Inversion with Prompt) techniques.
• Response Branch: Similarly applies corresponding initialization techniques. By jointly applying these techniques, the attacker can generate preliminary estimated versions of inputs and responses.

3.2 Phase 2: Patched Model Inversion (PMI)

After obtaining initial estimates, the authors further optimize the estimates through a novel Patched Model Inversion training method.

• This method aims to produce optimized embedding vectors.
• The final reconstructed sequence is extracted from these optimized vectors.
• This strategy is specifically optimized for long-sequence problems and significantly outperforms previous inversion methods.

Figure 2 Description: Figure 2 in the paper details the flowchart of the PIDI attack. The figure is divided into two main parts: the left side shows the “Dual-Sided Initialization (DSI)” phase, demonstrating how to initialize the input and response branches respectively; the right side shows the “Patched Model Inversion (PMI)” phase, demonstrating how to optimize embedding vectors through training to extract the final reconstructed sequence. The figure clearly explains how the attacker leverages activation information from both ends of the model to work collaboratively to maximize leakage risk.

4. Defense Method: ADMI

To address threats from both ends, the paper further proposes Adapter-based DualGuard with Mutual Information Defense (ADMI).

4.1 Core Mechanism

ADMI aims to provide strong empirical privacy protection while minimizing the impact on task performance. Its core consists of two parts:

1. Adapter-based local warmup strategy: By performing specific warmup on the model locally, it induces rapid changes in model parameters, thereby increasing the difficulty for attackers to reconstruct data.
2. Mutual Information Regularization: Introduces a mutual information minimization objective to encourage the model to learn privacy-preserving representations, reducing the amount of sensitive information contained in intermediate activations.

4.2 Advantages

Compared with existing defense methods (such as Differential Privacy, Sparsification, token-based perturbation, etc.), ADMI can simultaneously defend against attacks from both the model Head and Tail. The paper points out that existing DualGuard methods replace the model Body with lightweight projection layers during the warmup phase and freeze the Head, which may limit the degree of model adaptation. In contrast, ADMI better balances the Privacy-Utility Trade-off through the Adapter mechanism.

5. Experiments and Conclusion

The paper conducted extensive experiments on diverse tasks and models. The results show:

• Effectiveness of PIDI Attack: PIDI can significantly outperform existing inversion methods, successfully reconstructing both inputs and outputs from the Split-LLM setup.
• Effectiveness of ADMI Defense: ADMI can effectively resist PIDI and other state-of-the-art inversion attacks, maintaining good task performance while protecting privacy.

The authors have made the code public on GitHub (https://github.com/FLAIR-THU/VFLAIR-LLM) and implemented the related work based on the VFLAIR-LLM framework, providing an open-source benchmark for systematically evaluating privacy risks and defenses in Split-LLM.

6. Summary

This paper deeply explores the bidirectional data leakage problem in Split-LLM, filling the gap in previous research that primarily focused on input leakage while neglecting output leakage. By proposing the PIDI attack and ADMI defense, the authors not only reveal potential security risks but also provide an effective mitigation solution. This work has important reference value for the secure deployment of Split-LLM in privacy-sensitive domains.