Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions

Judged Useless, Queried Anyway. Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions. By Chubin Zhang and six others, from Nanyang Technological University, the National University of Singapore, and three other institutions. Published on the fifth of October. Its code repository is public and real: about twenty-five source files.

An agent whose tool keeps returning nothing useful ought to stop relying on it. The claim here is that tool-using agents largely do not, even when they themselves have said the results are useless. Judgment and action come apart. The agents recognise a failing source, and that recognition rarely becomes the reason they stop.

The setting is a retrieval environment where source failures are controlled, which lets the authors separate how an agent judges results from what it does next. The key measurement compares stopping at the same step after a longer run of results the agent judged useless, against a shorter run. If an agent stopped by the clock or by a deadline, that contrast would be zero. Only stopping driven by the evidence makes it move.

Where judgments are recorded, the seven agents tested call a failing source's results useless between ninety-seven and one hundred percent of the time. Yet most of them rarely stop on that judgment. Prompt cues change when agents stop, but not what they stop on. Permission to answer from memory, or a reasoning mode, can bring early stops whatever the evidence says. A stated budget moves the stops of the smaller models, those of seven to eight billion parameters, to the deadline. A stopping rule or a cost per call written into the prompt is followed at most partly.

One intervention works. When the harness enforces an integration step, making the agent answer after five consecutive results it judged useless, stopping finally tracks the evidence. That step raises success on failing sources for every model. It keeps the stopping point fixed when the budget doubles. And when the agent already states its judgments, it needs no extra judgment call. A pre-registered replication on three hundred fresh questions confirms both the dissociation and the rule's effect.

The evidence comes from one kind of environment: retrieval, with source failures the authors control. Seven agents are tested, and the headline rates of judging results useless hold where those judgments were recorded. The fix that works lives in the harness, not in the prompt, and its effect is reported for failing sources specifically. What is shown is a dissociation in this setting, and a rule that closes it there.

The practical reading is a division of labour. Agents are good at saying a source is useless, and poor at acting on what they say. Rather than asking the prompt to carry the decision, the work hands it to the harness, which turns a judgment the agent already made into the stop it did not make on its own.