Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents

Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents. By Nghi D. Q. Bui, from Google. Published on the twenty-sixth of October. No code repository is linked to it.

Challenge a language model agent on work it got right, and it will often abandon that work. The obvious fix fails in the other direction: tell the agent to stand firm, and it starts ignoring the mistakes it really made. The diagnosis offered here is about timing. Once an agent has read a complaint, it tends to discount the checks it had already done, so that it can agree with the user.

If reading the complaint is what undermines the agent's own evidence, then the evidence should be written down before the complaint arrives. That is Thinking-in-Bets, a method that needs no training. Before it reads any user feedback, the agent writes a short private record: its main claims, the checks that support them, and how confident it is. This record is called the Claim Checklist. When a challenge comes, the agent consults it, keeping verified work intact and spending its investigation on the parts that were uncertain or are newly under challenge.

The evaluation uses five Gemini models: three point five Flash-Lite, three point five Flash, three point six Flash, three point eight Flash, and four point zero Argon. They are run across eight benchmarks. For software engineering, SWE-Together, SWE-Interact, and DeepSWE. For professional reports, Agents' Last Exam. For academic reasoning, Humanity's Last Exam and SycEval. For financial analysis, Vals Finance. And for command-line tasks, Terminal-Bench.

A variant gates the record, so it is written only when the agent is uncertain. Another applies the idea without any user at all: list the key claims in a draft, and re-check the weakest one before submitting.

Recording claims before criticism nearly eliminates false reversals in patch review, and it protects report drafts that were already passing when the pushback against them is mistaken. A control isolates the timing. Writing the same record after reading the complaint recovers very little of the gain, so the content of the record is not enough on its own. Its timing matters.

In multi-turn coding, claim records written before replying substantially reduce unnecessary file edits after mistaken complaints. The gated version lowers the token overhead, and still fixes more real bugs than the baseline.

The last setting removes the user from the picture entirely. Even with no user feedback to resist, listing the key claims and re-checking the weakest one improves both answer accuracy and the quality of confidence estimates, across every model tested.

The method asks very little of the agent: a few lines of notes, written at the right moment. What it protects against is a specific failure, where an agent's own earlier reasoning loses its weight the instant someone disagrees. Committing to claims and their support in advance gives the agent something to check the complaint against, rather than something to give up.