EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
Yuling Shi, Zhensu Sun, Junsen Dong, Chengcheng Wan, David Lo, Xiaodong Gu
Shanghai Jiao Tong University · Singapore Management University · East China Normal University · Shanghai Innovation Institute
Published September 2, 2026 · on the feed September 3, 2026 · ▲ 113 · arXiv 2609.02783
The abstract
Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.
The Brief
2:21A short podcast summary of the paper. Written and voiced by AI.
Transcript
EarlyEval. Cheaper Agent Evaluation via Early Outcome Prediction. By Yuling Shi and five others, from Shanghai Jiao Tong University, Singapore Management University, and two other institutions. Published on the second of September. Its code repository is worked out: about one hundred and fifty source files, with a readme and a licence.
Evaluating language model agents is how their development gets steered, and it has become prohibitively expensive. A single pass of a frontier model over an agentic benchmark can cost anywhere from hundreds to thousands of dollars, and that price is paid again in every cycle of iterative development. Earlier efforts to cut it have centred on benchmark distillation, which shrinks the number of tasks evaluated. That leaves the cost of running each remaining task exactly where it was.
This work introduces early outcome prediction, a complementary axis of efficiency that cuts cost inside each task instead of across them. The key insight is that an agent's final outcome is often evident from its intermediate behaviour well before execution finishes. If you can already tell how a run will end, there is no need to watch it get there.
EarlyEval is a lightweight framework built on that idea. It trains a pair of LightGBM classifiers, one predicting success and one predicting failure. Their inputs are behavioural features, textual features, and features drawn from the reference solution. During a run, the agent is halted the moment either classifier crosses a calibrated confidence threshold. The per-step overhead this adds is negligible, which matters, since a check that costs as much as the step it watches would save nothing.
Three benchmarks are used: SWE-bench Verified, TerminalBench, and Toolathlon. Across them, EarlyEval can eliminate between thirteen and twenty-six percent of agent steps. It can also cut up to forty-four point one percent of input tokens, and up to twenty-nine point four percent of output tokens.
Those savings come at a prediction accuracy of between eighty-nine and ninety-seven percent. The quantity a developer actually reads off an evaluation, each agent's resolve rate, is perturbed by only one to two percentage points on average.
The two lines of work stack rather than compete. Benchmark distillation decides which tasks to run, and early outcome prediction decides when a running task has already told you its answer. Because they act on different things, the savings from one leave room for the savings from the other.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Evaluation bills keep climbing. Here is a team paying full price for runs whose ending they could have guessed.
You run evaluations for a team building LLM agents. One pass of a frontier model over an agentic benchmark costs hundreds to thousands of dollars, and you pay it again in every development cycle.
The road ahead
- 1 · The problem ~6 min Testing an AI agent on a benchmark is very expensive, and today's savings only cut the number of tasks, not the cost of each run.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min An agent's ending often shows early, so two classifiers watch each step and stop the run once either is confident.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build features, train success and failure classifiers, check them after every step, and halt at the first confident verdict.
- 6 · The experiment ~6 min You’ll guess: Halting agent runs early saves steps and tokens. How much did it shift each agent's resolve rate on average?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.