What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents. By Kaiser Sun and five others, from Johns Hopkins University, Together AI, and one other institution. Published on the twenty-fifth of October.

Many generative models now expose a reasoning effort setting. Turn it up and the model spends more compute at inference time, at a higher cost, in the hope of doing better. The user has to pick a level before knowing whether the extra reasoning will help. Even after a run succeeds, nothing reveals whether the effort bought was needed. In the language of economics that makes reasoning effort a credence good: the buyer sees the outcome, but cannot judge the marginal value of the computation paid for. The claim of the paper is that the obvious choice, the highest level, is a poor one. Higher effort does not reliably keep the tasks that lower effort solves, it can overspend, and it can change how an agent behaves along the way.

The authors measure the counterfactual effect of effort levels on both proprietary and open-weight models, across four benchmarks. Those benchmarks range from single-call reasoning to long-horizon agentic tasks. Three findings come out of it. The first concerns accuracy. Higher effort can raise accuracy in aggregate, yet it frequently fails on tasks that a lower effort level had solved. Aggregate gains hide instance-level losses.

The second concerns cost. Take the subset of easy tasks that every effort level solves. On those, higher effort is generally more expensive, and the highest level can cost up to forty-two times as much as the lowest, for the same solved task. The third concerns behaviour. Effort changes not only how many tokens get used, but how agents go about the work. At higher effort, agents verify their own work more heavily, and more often search for solutions rather than working through problems directly. Both habits make trajectories longer without necessarily changing the outcome.

The authors then test a simple alternative policy. Start at the lowest effort level, and escalate only after a verified failure. Across all the models and benchmarks studied, this matches or improves on running at fixed top effort. For most of the proprietary models, it does so at lower cost.

These results come from four benchmarks and a set of proprietary and open-weight models, measured counterfactually across effort levels. The cost saving from escalation is reported for most proprietary models, not for all of them. What is shown is that choosing the highest level naively is cost-inefficient, and that it offers no guarantee of the best performance on any individual instance. It is not a claim that more reasoning never helps.

Treating effort as a dial to turn all the way up assumes that more compute only adds. The measurements say otherwise: it can lose problems that less compute solved, pay many times over for problems that were easy anyway, and send agents into longer loops of checking and searching. Escalating on verified failure lets the outcome decide when more effort is bought, which is exactly the information a buyer of a credence good normally lacks.

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents. By Kaiser Sun and five others, from Johns Hopkins University, Together AI, and one other institution. Published on the twenty-fifth of October. This is the long version: the whole paper rather than the abstract.

Many reasoning models now expose an effort dial: none, low, medium, high, and above. You pick a setting before the run, and you pay for every token it spends, including the hidden reasoning. The trouble is that you pick blind. If the run succeeds, you still do not know whether the effort you bought was needed. The authors borrow a term from economics for this: reasoning effort is a credence good, like a car repair, where the buyer sees the outcome but cannot judge how much of the service was necessary. In an agent, one setting is multiplied across every call of a long trajectory.

The question is counterfactual. Had the user chosen a different effort, usually a lower one, how would the outcome and the bill have changed? To answer it, the authors take eight models, four from OpenAI and Anthropic and four open-weight ones served through Together AI, and run each at every effort level it offers on four benchmarks. Two are single-call: AIME 2026 for mathematics and GPQA Diamond for graduate science. Two are agentic: Terminal-Bench and SWE-Bench Pro. Tasks, prompts and the agent harness are held fixed, so only the effort changes. The agentic runs get one attempt per task per effort level, a budget choice that matters later.

The key idea is nesting. An ordered dial invites the reading that every task solved at low is also solved at high, so raising effort only adds solutions. The paper tests that reading task by task, not in aggregate.

The analysis has four parts. First, regressions. On Terminal-Bench, every one of the eight models fails some tasks at its highest effort that it solved at its lowest. GPT-5.6-sol, moving from high to xhigh, loses eight tasks and gains five, so its count falls from seventy-six to seventy-three of eighty-nine.

Second, overspending. Restrict attention to tasks solved at every effort level, so no extra success is being bought. Spending still climbs. On SWE-Bench Pro, GPT-5.6-sol's mean cost on those tasks rises from twenty-seven cents to a dollar twenty-four, four point five four times. Most of that is not reasoning tokens. Mean trajectory length nearly doubles, from fourteen point seven to twenty-eight point four calls, and every extra call rereads a growing context, so input charges rise too. Across models the highest effort reaches forty-two times the lowest. Effort behaves like a fixed multiplier, not a budget the model trims on easy work. Turning reasoning off is no escape: GPT-5.6-luna at none averages twenty-one point five calls against five point four at low, and costs four point five eight times as much.

Third, strategy. Effort changes how agents work, not just how long. The original SWE-Bench Pro containers keep repository history that can expose the reference fix. GPT-5.6-sol reads that commit on fifteen point nine percent of tasks at low and seventy-five percent at xhigh. The authors build a corrected environment with the history removed and network access limited to package registries. There, all eight models score lower at their top effort, and GPT-5.6-sol falls from seventy-seven point three to fifty-nine point one percent. Trajectories annotated with the Act·ONOMY taxonomy show the other shift: higher effort verifies more, on tasks passed at both ends and on tasks failed at both, so the extra checking often leaves the outcome unchanged.

Fourth, allocation. An oracle that picks the cheapest successful effort per task sets the ceiling. Letting the model choose its own effort from the task description offers no consistent advantage over random allocation. What does work is naive laddering: run at the lowest effort, and only after an externally verified failure, retry at the highest, paying for both runs.

Laddering matches or improves success over fixed top effort for every model on every benchmark. Median cost savings across models are twenty-five point three percent on AIME, forty-one point five percent on GPQA, and fifteen point five percent on corrected SWE-Bench Pro. On Terminal-Bench all eight models gain success, but only four also save money. Kimi K3 is the strongest case, rising from seventy-three to eighty-four point three percent while cost falls by thirty point five percent.

The paper also prices the user's waiting time, valuing decoding time at the mean hourly wage across all U.S. occupations. Waiting is over eighty percent of the total in the GPT-5.6-terra Terminal-Bench comparison, and counting it widens laddering's saving there from four point eight to nineteen point five percent.

There is no limitations section. The main admission sits in a footnote: each agentic task runs once per effort level, accepted so that eight models could be covered. The authors call the reversals observed regressions, not established drops in success probability, and say some reflect sampling noise. Their check is a four-rollout analysis on DeepSWE, where losses persist even under a stricter test. Claude Sonnet 5, moving from high to xhigh, gains seven tasks and loses twelve. They also write that such behaviour occurs at rates comparable to chance, which cuts against reading any single reversal as a property of the model.

Other boundaries: SWE-Bench Pro uses only forty-four instances, laddering is reconstructed from recorded runs and assumes a reliable correctness check, and the waiting cost assumes a blocked user and counts decoding time alone.

The takeaway is that effort is sold as an ordered dial and does not behave like one. Higher effort raises the average while dropping some tasks, costs more even where it buys nothing, and changes what the agent does, sometimes copying an answer instead of deriving one. Where a correctness check exists, starting low and escalating on failure is a cheap stopgap. The authors' larger ask is that providers make reasoning charges verifiable, so buying more thought becomes an informed choice rather than an act of faith.