What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation
Daisuke Kikuta
NTT, Inc.
Published September 3, 2026 · on the feed September 8, 2026 · ▲ 6 · arXiv 2609.03254
The abstract
Large Language Models (LLMs) often help users generate artifacts through iterative cycles of generation and revision in conversation. A challenge here is that, when users specify only a local change during revision, LLMs must instead identify the relevant dependencies and propagate the revision to all affected parts of the artifact. This paper studies this ability of LLMs on conversationally generated artifacts, where the artifact context and its dependencies may be embedded in the conversation history. Toward practical use, we also explore cost-effective test-time compute for this new setting. Specifically, we introduce a new benchmark for this setting, and evaluate nine revision methods, including sequential reflection and parallel sampling variants, using gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b on the benchmark. The results show that baselines achieve accuracies of 68.3--93%, and the most cost-effective method is selecting from three parallel samples using either LLM-based or medoid selection, which improves accuracy by 2.2--9.7%. Our code and dataset are available at https://github.com/ntt-dkiku/llm-revision-propagation.
The Brief
2:05A short podcast summary of the paper. Written and voiced by AI.
Transcript
What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation. By Daisuke Kikuta, from N T T. Published on the third of September. Its code repository is public and real: about thirty source files.
People increasingly build artifacts with a language model through conversation: generate something, ask for a change, generate again. The catch is that a user often asks for only a local change. The model has to work out for itself which other parts depend on that change and carry the revision through to all of them. And because the artifact was built up over a conversation, its context and dependencies may be scattered through the conversation history rather than sitting in one place.
The ability under study is revision propagation: given a local edit, does the model find and fix everything else that now needs fixing. Alongside it the authors study cost, asking which ways of spending test-time compute on this new setting are worth what they cost in practice.
A new benchmark was built for the setting. On it the authors evaluate nine revision methods, including sequential reflection and parallel sampling variants. The models are gpt-oss at twenty billion and at one hundred and twenty billion parameters, gpt-5.4-mini, and qwen3.5 at three sizes: nine billion, twenty-seven billion, and one hundred and twenty-two billion.
Baselines reach accuracies from sixty-eight point three percent up to ninety-three percent. The most cost-effective method is to draw three parallel samples and select one, either with an LLM as the selector or by medoid selection. That improves accuracy by between two point two and nine point seven percent. Code and dataset are released.
The question in the title is the one the benchmark asks every model: after the change you were told to make, what else needs fixing. The baselines already answer it much of the time, but not all of it, and that remaining share is where test-time compute has to earn its cost. The method that gave the best return for its cost is modest in form. Three samples, and a way of choosing among them, rather than sequential reflection.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
One small request, and a dozen quiet consequences. Can a model find them all?
You have spent an hour with a chat model building a day trip plan as a JSON file: stops, times, number of people, budget. Now you say just one thing: there will be three of us.
The road ahead
- 1 · The problem ~7 min When you ask a chat model for one small change, it must find on its own every other part that change affects.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Draw a few answers in parallel, then pick one whole answer, by asking the model or taking the most typical one, instead of merging edits.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min An artifact is built over a chat, a local revision is requested, three samples are drawn, and one is selected by an LLM or the medoid.
- 6 · The experiment ~6 min You’ll guess: Of the nine revision methods tried, which way of spending test-time compute was the most cost-effective?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.