Reading Notes

Personal reading notes. Original paper: arXiv:2606.13802v1.

Reading Notes: A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets

Paper ID: 2606.13802v1
Title: A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets
Authors: Tejas Agrawal, Vu Le, Sumit Gulwani, Gust Verbruggen
Source: arXiv (2026)

1. Research Background and Motivation

The authors point out that predictive code completion significantly accelerates developer efficiency. However, in the domain of spreadsheets, despite their usage frequency being far higher than code, similar autocomplete features are almost non-existent. Existing spreadsheet assistants focus primarily on specific scenarios, such as FlashFill for automatically filling columns, or suggesting formulas only after the user explicitly signals intent. For routine editing and repetitive patterns, the cost of invoking auxiliary tools is often higher than direct manual operation, leading users to prefer manual editing.

The paper argues that there is currently a lack of work on generalized action suggestions for spreadsheets, mainly facing two challenges:

1. Lack of Edit History Data: Unlike code, which possesses detailed version history, public spreadsheet corpora typically contain only high-level evolution data, lacking fine-grained action sequences.
2. Complex Action Space: Spreadsheet actions involve spatial (cell location), temporal (operation order), and composite (formatting, data, merging, etc.) dimensions, and there is no clear prediction trigger timing.

To fill this gap, the authors introduce a benchmark and evaluation framework for assessing systems that observe user action sequences and predict future actions.

2. Dataset Construction Method

To address the lack of edit history, the authors manually curated 52 action sequences, totaling 12,000 actions, to reconstruct spreadsheets from public corpora. The dataset construction employs a three-stage pipeline aimed at reducing manual labeling workload and introducing natural variability:

1. Symbolic Cold-Start Generation: Given a target workbook, the system decomposes its final state into cell-level operations and merges adjacent identical operations into range actions (e.g., merging font(A1, bold) and font(A2, bold) into font(A1:A2, bold)). To introduce diversity, the system samples from a preference space to control merge strategies, region orders, and formatting orders.

2. LLM-Assisted Refinement: Utilizing an LLM-based “judge-editor” loop as an auxiliary step, the system identifies and corrects obviously unnatural patterns in the symbolic sequences. For example, it merges scattered cell formatting into range operations, removes stray formatting in unused areas, and adjusts sequences to follow more natural patterns.

3. Human Annotation: Human annotators perform a final round of checks to correct remaining unnatural subsequences, ensuring all revisions achieve the target workbook state.

The final dataset contains 52 high-quality workbook creation trajectories, with each trajectory containing between 35 and 821 steps, averaging 229 steps, for a total of 11,907 steps. Supported operation types include inputting data, merging cells, setting formats, filling colors, font settings, border settings, alignment, pasting, and autofill, among others (as shown in Table 1 of the paper).

3. Online Evaluation Framework

To address the complexity of the action space and the lack of clear trigger timing, the authors propose an Online Evaluation method, rather than a static “given x predict y” offline evaluation.

In this framework, the system under evaluation is given the past $n$ actions at each iteration step and predicts zero or more subsequent actions. Based on the attributes of the prediction, the system decides to accept or reject the prediction:

• If Accepted: The future action sequence is updated. For example, operations successfully completed in the prediction are removed, and actions to undo erroneous operations are added.
• If Rejected: The next user action is added to the history, and the next iteration begins with $n+1$ actions.

This process repeats until there are no remaining actions (the final spreadsheet is obtained) or a threshold is reached. The authors note that this online evaluation has three advantages over fixed-timestep offline evaluation:

1. Suggestions change the state of the model and the user’s continued operation; errors may accumulate, and subsequent predictions depend on prior predictions.
2. The model cannot repeatedly suggest the same simple action and still appear accurate.
3. The system is tested on its ability to correct or fix its prior suggestions.

The paper illustrates one step of online evaluation in a synthetic example via Figure 1.

As shown in Figure 1, historical operations produce the current state, and remaining future operations will produce the target state. The system predicts three actions based on history and current state: two border actions match the future, while one fill action targets the wrong cell. At the cell level, this yields a precision of 7/8. If the user accepts the prediction, since the prediction satisfies two border operations but requires adding an operation to undo the erroneous fill, the user saves one action overall. If rejected, the future remains unchanged, and the evaluation continues.

4. Baseline Models and Experimental Insights

The authors use various baseline predictors to validate the benchmark and framework, including Zero-shot LLMs, Fine-tuned Small Language Models (SLMs), and classic machine learning models. Through metrics at different levels (per simulation, per prediction, per action), the authors analyze multiple attributes revealed by the benchmark:

• Relationship between Model Capability and Action Savings: More powerful models save more actions. For example, the reasoning-capable GPT-5 saves 33% of actions, while GPT-5 mini saves 18%. Notably, the performance of a fine-tuned 360 million parameter model is comparable to GPT-5 (both at 27%), indicating that this task is learnable.
• Importance of Acceptance Heuristics: Low-precision acceptance heuristics lead to negative savings (-19%), confirming that an abstention mechanism based on net user benefit is crucial.
• Trigger Timing and Context: Predicting after every action saves more actions (27%), while predicting after every four actions saves only 17%. This suggests that studying good trigger mechanisms or cheaper models is worthwhile.
• Other Attributes: The paper also analyzes attributes of saved actions, False Positives, efficiency, the impact of user profiles, and the impact of context.

5. Summary and Contributions

The main contribution of this paper lies in filling the gap in the field of predictive autocomplete for spreadsheets. The authors not only constructed a high-quality benchmark dataset containing 52 trajectories and nearly 12,000 actions but also proposed an end-to-end online evaluation framework that simulates user acceptance behavior.

Through this framework, researchers can calculate metrics that reflect real-world utility to improve predictive assistants. The paper demonstrates how to use these metrics to guide model optimization, such as finding that fine-tuned small models can achieve the effects of large models, and emphasizing the importance of prediction acceptance strategies for the final user experience. Related code and artifacts have been open-sourced to promote the development of spreadsheet productivity tools.