Reading Notes

Personal reading notes. Original paper: arXiv:2606.08656v1.

arxiv_id: 2606.08656v1

Paper Reading Note: From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

1. Research Background and Motivation

As the deployment of Large Language Model (LLM) agents in long-running scenarios increases, Test-Time Learning (TTL) capabilities become crucial. TTL refers to the agent’s ability to continuously improve performance over a series of interactions during deployment by leveraging accumulated experience.

The paper points out that existing TTL methods typically rely on explicit memory updates, updating memory after each interaction to guide future decisions. However, most existing methods (such as Reflexion, ExpeL, Dynamic Cheatsheet, etc.) rely on manually designed prompt rules or heuristic mechanisms to update memory. The authors argue that this approach struggles to align memory updates with downstream goals over multi-step time horizons, resulting in even powerful instruction-following LLMs finding it difficult to continuously improve in repeated interactions relying solely on heuristic mechanisms.

To bridge this gap, the authors propose treating memory update as a trainable capability rather than a static prompt engineering task.

2. Core Method: MemoPilot

The authors propose MemoPilot, a plug-in memory copilot designed to enhance the performance of frozen LLMs in sequential interactions through end-to-end training of the memory update process.

2.1 Framework Overview

The core idea of MemoPilot is to treat memory as an artifact that evolves continuously over multiple interactions. The framework consists of a trainable memory model $G_\theta$ and a frozen player model $\pi$. The memory model is responsible for extracting strategic insights from interaction trajectories and generating text memory to guide subsequent games; meanwhile, the player model itself remains stateless across games, making decisions based solely on the current memory.

The paper illustrates the overall framework of MemoPilot in Figure 2. As shown in the figure, the trainable memory model $G_\theta$ iteratively updates the memory state $m_t$ based on interaction trajectories and provides it as guidance to the plug-and-play frozen player $\pi$. All cross-game learning relies on the continuously evolving memory, while $\pi$ remains unchanged.

2.2 Multi-turn Memory Generation as MDP

The authors formalize multi-turn memory update as a Markov Decision Process (MDP).

• State Space: Composed of observation tuples $s_t = (e_t, m_{t-1})$, where $e_t$ is the latest game trajectory and $m_{t-1}$ is the previous memory.
• Action Space: Text memory space, where the generator samples $m_t \sim G_\theta(\cdot | s_t)$.
• Reward Function: Returns the observed game outcome $r_t$.

The training objective is to maximize cumulative performance over multi-turn interactions. Since the first game usually serves as initial exploration (without learning guidance), the authors define the episode return as the sum of rewards calculated starting from the memory-guided games.

2.3 Training Strategy based on Multi-turn GRPO

To optimize the memory update policy, the authors employ Multi-turn GRPO (Group Relative Policy Optimization) for end-to-end optimization. The paper proposes a new training recipe containing two key innovations:

1. Turn-wise reward signal: Provides explicit reward feedback for each turn of interaction.
2. Context-independent, turn-level advantage estimation: Calculates advantage across multiple rollouts to achieve finer-grained credit assignment and stabilize training in multi-turn settings.

The paper illustrates the multi-turn GRPO training process for memory update in Figure 3. The figure depicts the optimization process utilizing single-step (next game) agent rewards and turn-wise advantages. Each rollout contains multiple game turns; the memory model is updated based on accumulated trajectories and advantage signals, thereby learning to iteratively optimize memory quality within a “hypothesis-verification” loop.

3. Experimental Evaluation

The authors evaluate MemoPilot on two strategic game testbeds: Multi-turn Rock-Paper-Scissors (RPS) and Limit Texas Hold’em (LHE). These environments were chosen as testbeds because they fit the TTL setup, featuring learnable opponent behavior structures, controllability, and measurable rewards.

3.1 Test-Time Learning Dynamics

The paper presents the test-time learning dynamics of MemoPilot in Limit Texas Hold’em (LHE) in Figure 1, comparing it with baseline memory models. Although the figure does not directly provide a file reference, according to the text description, the figure shows the cumulative performance of two frozen players (Qwen2.5-14B-Instruct and Qwen3-235B-A22B) across consecutive games. Results show that the MemoPilot model can consistently provide higher cumulative performance and improve rapidly within a few games. Even when zero-shot generalizing to stronger frozen players, MemoPilot demonstrates significant advantages.

3.2 Performance Comparison

Experimental results indicate that after integrating the MemoPilot plugin with frozen players, test-time learning performance significantly outperforms strong baselines.

• Elo Rating: MemoPilot ranks first in both games, reaching 1762 in LHE and 1590 in RPS.
• Comparison Models: MemoPilot surpasses all baseline memory methods as well as proprietary models, including DeepSeek-V3.2.

4. Summary and Contributions

The main contributions of the paper are summarized as follows:

1. Proposed MemoPilot, a plug-in memory copilot that enhances the test-time learning behavior of frozen LLM players in repeated interactions through end-to-end training of the memory update process.
2. Introduced a multi-turn GRPO training recipe for memory update, including turn-wise rewards and turn-wise advantage estimation, achieving stable credit assignment in multi-turn test-time learning rollouts.
3. Validated MemoPilot on controlled game testbeds, demonstrating consistent gains in test-time learning.

This work emphasizes the importance of treating memory update as a trainable decision process, providing new solutions for the adaptive capabilities of LLM agents in long-horizon tasks.