AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

AgentJudgeBench. A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling. By Abhigya Verma and three others, from ServiceNow AI. Published on the twenty-seventh of August. No code repository is linked to it.

LLM judges are now a common way to evaluate agentic tool-calling systems: one model does the work, and another model decides whether the work was right. Tool calling, though, is not open-ended text. Its workflows are structured and dependency-driven, with each call relying on what came before, and the whole run forms a directed acyclic graph. How reliable a judge is on that kind of work has gone largely unexamined. The broader study of LLM-as-a-judge mostly concerns open-ended text and preference evaluation, which is a different problem. AgentJudgeBench is presented as the first benchmark to study judge reliability specifically for tool-calling over workflow DAGs.

The quantity under test is alignment: how closely a judge's verdict agrees with a reference. Every judgment is collected under paired conditions. In one, the judge sees the ground truth. In the other, it does not. That pairing lets the benchmark ask a question that is easy to assume away, namely whether handing a judge the answer actually helps it judge.

The benchmark holds three thousand eight hundred and eight instances. They span six DAG topologies and three difficulty tiers. The responses come from five generators: open-weight models between three billion and seventy billion parameters, plus GPT-5.4. Six judges grade them, ranging from twenty billion parameters up to frontier scale. On top of that, the study tests mitigation strategies, namely chain-of-thought reasoning, judge temperature, and structured evaluation rubrics, and it runs a human validation study alongside the programmatic reference.

The central result is a ceiling. Judge alignment degrades monotonically as tasks get harder, and it degrades one point five times faster without ground truth. On hard queries without ground truth, all six judges converge to a narrow band, between seventy-seven and eighty-two percent, regardless of scale. The authors read this as a structural ceiling, driven primarily by task difficulty, that model capacity alone cannot overcome. Its height is partly prompt-dependent for weaker generators.

Ground truth is not uniformly beneficial either. It reduces alignment for GPT-5.4 by one point five percentage points, and for Gemini-2.5-Pro by three point nine, a pattern consistent with over-anchoring. Among the mitigations, chain-of-thought and judge temperature have negligible effect. Structured rubrics improve alignment by up to six point five percentage points, but do not generalize uniformly across judge-generator pairs. As for which judge to use: with ground truth, QwQ-32B best matches the programmatic reference, while the human study picks out GPT-OSS-120B as the most human-aligned. Without ground truth, frontier judges lead only marginally, inside the shared ceiling.

The authors turn all of this into practical guidelines for reliable evaluation in agentic systems, and the frame around those guidelines is sober. On hard, dependency-driven tasks, a bigger judge buys very little. The limit sits mostly in the difficulty of the task itself, and the cheap levers, a reasoning trace or a different temperature, barely move alignment at all.