Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

Good Judgment, Bad Advice. Decision-Model Controllers Raise Agent Cost Without Improving Success. By Naman Vats. Published on the seventeenth of October. Its code repository is worked out: about forty-five source files, with a readme and a licence.

Agent scaffolds increasingly hand small decisions to cheaper models: whether to stop, whether to verify, whether to change approach. This study asks whether a typed decision model called Jev 1.13, consulted only at deterministic decision points, can make a Claude Sonnet 5.5 terminal agent cheaper without making it worse. The answer it reports is no. The controller raised cost and left success where it was. Its judgments were informative in isolation, yet as interventions they did not help.

The study was pre-specified and run on Terminal-Bench 2.1. One agent loop is held fixed, and only the controller varies across five arms: no controller, deterministic rules, a small language model, Claude Haiku 5.5, answering the same questions, Jev itself, and a placebo that samples Jev's development-phase choice frequencies at the same moments.

On twenty-one held-out tasks, comprising three hundred and fifteen trials, Jev raised cost per attempt by nineteen percent. As a geometric-mean ratio that is one point one nine, with a ninety-five percent confidence interval from one point zero two to one point three six. The pass rate was identical, seventy-three percent in both arms. Jev was cheaper than the small language model controller, at a ratio of zero point eight one, but twenty-two percent more expensive than the placebo. No pass-rate difference between any of the arms was detectable.

Offline, Jev ranked finish requests better than the alternatives, with an AUROC of zero point seven eight, against zero point six two for the rules and zero point six for the small model. But it was poorly calibrated. Its mean probability that a run was complete was zero point five, when eighty-seven percent of those runs had in fact passed. So it asked the agent to verify work that was already correct. And no controller ever chose to abandon a run.

The result is bounded by its design: one terminal agent, one benchmark, a held-out set of twenty-one tasks, and one particular decision model. Within that frame the gap is between ranking and acting. Informative decision signals did not become useful interventions, and the author discusses what would be required for them to do so.

The title compresses it well. Jev could tell finished runs from unfinished ones better than its rivals could, and still cost more, because its doubt was pitched too high and every doubt bought another round of verification on work that had already passed.