VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

VeriHarness. Scaling Agentic Verification for Long-Horizon Tasks. By Caiqi Zhang and six others, from the University of Cambridge and Google Cloud AI Research. Published on the first of October. Its code repository is worked out: about forty-five source files, with a readme and a licence.

As LLM agents take on more complex, long-horizon tasks, verifying their outputs gets harder. The question here is how to strengthen verification with a fixed base model, and without reference answers or grading rubrics at test time. Repeated sampling helps in one way: multiple rollouts can contain complementary correct claims, each getting right something another missed. But that only pays off if something can reliably decide which claims to trust. Without a reference answer or a rubric to grade against, the verifier has nothing to lean on except the rollouts themselves and whatever evidence it can find in the task's environment.

An early observation shapes the design. Disagreement between rollouts often exposes correct alternatives, while consensus can conceal errors. Agreement, in other words, is not the same thing as correctness. That cuts against the obvious strategy of trusting whatever most rollouts say. When rollouts differ, the minority may be the one that got it right, and when they all agree, they may all share the same mistake. VeriHarness responds by turning the same underlying LLM a generator uses into an agentic verifier. It gives that model a workspace, evidence tools, and reusable verification skills.

Two roles split the work, one for each half of the observation. A disagreement resolver checks competing claims against environmental evidence. A consensus challenger takes the claims the rollouts share, tests them, and searches for requirements that were omitted. Their findings then guide both the selection of the final artifact and its revision. The resolver handles the case where rollouts conflict, settling it with evidence rather than with a vote. The challenger handles the case where they agree, treating that agreement as a claim to be tested rather than as a verdict. And because the verifier is the generator's own model, the gains come from the harness around it rather than from a stronger judge.

Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision improves average performance further. With Gemini three point five Flash, the gain over a single rollout reaches six point two points. With Claude Opus four point eight, it reaches six point four points.

The verification skills can also self-improve from failure feedback. And the authors release the full pool of roughly twenty-six thousand rollouts across all five benchmarks and both models, produced at a cost of over one hundred thousand dollars, to support future research on agentic verification.

The premise underneath is simple to state. Treat consensus with suspicion and disagreement as information, then send a verifier with tools into the workspace to check both against the evidence there. The authors present this as a way to scale verification for long-horizon agents while the base model stays exactly as it was.