Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

Same Simulation, Different Evidence. Where Retrieved Tool Output Costs LLM Agents Accuracy. By Ke Zhang and four others, from the University of California, Riverside and the University of California, Irvine. Published on the nineteenth of October. Its code repository is public but slight: two source files.

An agent built on a language model can compute the right answer and still give the wrong one. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads when it needs them. The finding here is that, for some models, this costs accuracy, and that the cost comes not from computing worse but from answering from the wrong evidence. The authors call the failure evidence substitution.

The test uses a real scientific tool, PHREEQC, a widely used geochemistry simulator from the USGS. Its reports reach eight hundred and forty-six point eight thousand characters, and because it is deterministic, every printed value can be traced. Agents use it to answer one hundred and fifty-eight multiple-choice items, each verified by execution.

When agents read the simulator's report through a section index instead of in context, GPT-5.1 loses eleven point three points of accuracy and GPT-5.2 loses seven point nine. Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost.

Re-running all eleven thousand and seven logged simulator calls shows what happened. Both GPT models compute the keyed value about as often as before. What changes is the evidence they answer from. Under retrieval they often skip the report, answer from their own output files at higher precision, and misround to a neighbouring option. Holding the simulation fixed, retrieval still costs GPT-5.2 nine point four points.

GPT-5.4 shows how a net figure can hide the effect. It produces correct simulations more often, and that masks an evidence-use loss of about ten points, leaving a net change indistinguishable from zero.

Some fixes help and one does not. Naming the authoritative output raises accuracy over plain retrieval by between six point three and seven point nine points, for the three GPT models. For GPT-5.1 and GPT-5.2, inlining outputs that fit works as well. Asking for correct rounding does not.

The evidence comes from a single simulator and a set of multiple-choice items. The cost is model-dependent: present for the GPT models studied, not detected for the two Claude models, and hidden in the net score for GPT-5.4. The finding concerns harnesses that move tool output out of context, read through retrieval such as a section index.

The authors' recommendation is that context-management choices in agent harnesses be judged by where accuracy is lost, not only by final accuracy. A model that ran the right simulation and then trusted its own file over the report has made a different mistake from one that simulated wrongly, and the two call for different remedies. Final accuracy alone would not tell them apart.