Rethinking Verification in Agent Harnesses

Rethinking Verification in Agent Harnesses. By Zhengxin Zhang and five others, from Cornell University and Alibaba Group. Published on the twenty-fifth of October. No code repository is linked to it.

An agent harness lets a large language model act through tools, observe what happens, and work iteratively toward a goal. Verification is the effort that checks the agent's work and revises it to raise output quality. There is little guidance on how a harness should organise that effort, and this study treats verification as an axis of test-time scaling. It asks three questions: when to intervene, who does the checking, and how a check turns into a revision.

The headline claim is that independent review by a critic gives a more favourable trade between accuracy and cost than the alternatives built into the agent's own loop.

The framework covers four layers of an agent's lifecycle. Initial instructions. Reminders in the middle of a turn. Completion gates. And independent review. Seventeen designs are compared across seven frontier coding models, on Terminal-Bench, abbreviated TB, and on SWE-bench Verified, abbreviated SWE. The authors also introduce VisualBench, for expert evaluation of visual and interactive artifacts.

The first two layers behave modestly. Initial instructions and reminders during execution give average gains that are small and depend on the model, though they do not substantially raise average output token use. Completion gates can give larger gains, but may raise cost substantially.

Independent review is where the balance shifts. On TB it improves average accuracy by five point nine to nine point five percentage points, at between one point one three and one point two five times the baseline cost. On SWE the gain is three point nine to six point six points, at between one point zero zero and one point zero four times baseline cost.

On VisualBench, independent review earns net preferences of thirteen point two to twenty-three point eight points over the baseline across its four designs, ahead of the evaluated designs at layers zero through two. Adding further review rounds, or more independent critics, lifts average TB accuracy by up to three point five percentage points more.

The numbers describe averages over the models and benchmarks tested: seven frontier coding models, two coding benchmarks, and VisualBench. The gains from instructions and reminders are explicitly model-dependent, so an average hides variation between models. Completion gates are not dismissed. They can deliver larger gains, and the caveat attached to them is cost rather than accuracy. The extra improvement from more rounds or more critics is reported on TB. The study offers empirical guidance about design choices inside a harness, not a rule that holds for every model or task.

Put plainly, the place to spend verification effort may not be inside the agent's own loop at all. A separate reviewer, looking at finished work, bought more accuracy per unit of cost here than instructing, reminding, or gating the agent did. The authors read that as support for independent review as a practical way to scale at test time.