Evidence-First Agentic Verification

Evidence-First Agentic Verification. By Reza Sanayei and nine others, from Scale AI and the University of Arizona. Published on the thirteenth of October. Its code repository is worked out: about sixty-five source files, with a readme and a licence.

Long-horizon agentic tasks that are only partly verifiable are often graded with rubrics. The evidence needed to grade them, though, can be scattered across two places: the state of the environment, and the trace of what the agent did. A judge may overlook that evidence, or accept claims nobody supported, including the agent's own report that it succeeded. When rubric verdicts are used as rewards for reinforcement learning, those errors do not stay contained. They can reinforce exactly the behaviour no one wanted.

Sauron is an evidence-first agentic verifier. It investigates both the state and the trace, and it relies on a claim only after independently validating that claim against its source. An agent saying it finished is a claim like any other, to be checked rather than believed.

Two choices keep the investigation affordable. Queries are bounded, and related criteria are investigated jointly, which cuts context use and redundant work. And every verdict carries an auditable receipt of the evidence that supports it, so the grade can be traced back.

The main test uses two thousand five hundred and eighty-six human-labeled criteria, drawn from one hundred personal-assistant episodes. Sauron gets the highest F1 of five verifiers on all eight backbones tested, while costing less than Gandalf, the strongest baseline. With GPT-5.6 Sol as the backbone, Sauron reaches an F1 of zero point nine six against Gandalf's zero point nine two, at twenty-three percent lower cost, and the margin is wider on process criteria.

A paired fault-injection suite covers thirteen families of fault, several of them simulating reward hacking. Sauron detects ninety-seven percent of the injected failures while flagging only two percent of clean controls, and it beats the baselines even when they are handed the full trace. Finally, on TRAIL, a benchmark for detecting and localizing real agent errors, Sauron leads both Gandalf and TRAIL's own judge on all three quality metrics.

The stakes come from where the verdict goes. A grader that takes an agent at its word becomes, once it is a reward, a teacher of saying the right thing rather than doing it. Sauron's answer is procedural: find the evidence first, check each claim against its source, and attach the receipt, and the results on reward-hacking faults are where that procedure is put most directly to the test.