AgentProofBench: Benchmarking Reusable Verifiers for Web Agents

AgentProofBench. Benchmarking Reusable Verifiers for Web Agents. By Jingjie Ning and three others, from Carnegie Mellon University. Published on the twenty-fifth of October. Its code repository is public but slight: two source files.

A verifier for a Web agent is only reusable if its rule keeps following requirements as they change, and keeps supporting the execution that comes after it. Testing a rule on its own says little about the second part. What matters is whether a test of the rule tells you anything about what happens when it is put to work.

AgentProofBench connects the two. It pairs tests of a frozen acceptance rule with execution that starts from identical saved openings, so a failure in a test can be traced to its consequence in a run. The benchmark measures reusable verification at two levels: the requirement level and the policy level.

The resource spans ninety-seven tasks in seventy-nine families, across four Web services. Experiments use DeepSeek Flash and DeepSeek Pro, and establish two linked measurements. The release includes the instructions, states, programs, reference rules, and trajectories, which is what lets each test failure be connected to what it caused during execution.

The first measurement covers forty-eight families. Complete-panel and decision-replay tests retain eighty-two of ninety-six source programs for Flash, and seventy-nine of ninety-six for Pro. The paired reuse-rebuild BA intervals run from minus five point four seven to plus one point five seven percentage points for Flash, and from minus ten point one six to zero for Pro.

The second covers twenty-two execution families. Here, whether the panel is correct in the deployment state separates program assignments whose two saved openings are both delivered correctly. With error-free panels, that holds for thirty-two of thirty-three assignments. With panel errors, it holds for one of eleven. Every reference-defined program-gate regression, and every withheld delivery, falls in that second group. Under the generic feedback tested, neutral continuation attains the highest rate of correct delivery, which gives a concrete baseline for measuring what a gate adds.

The thread linking the two measurements is that a panel's correctness is not an isolated score. Where panels were error-free, delivery almost always went right. Where they had errors, the failures concentrated, and the benchmark keeps the trail that connects one to the other.