JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

JEV-as-a-Judge. Accept When Confident, Escalate When Unsure. By Yubo Li and three others, from Carnegie Mellon University. Published on the twenty-second of September. No code repository is linked to it.

LLM-as-a-judge makes evaluation possible across very different tasks, but at scale two things become critical: what each judgment costs, and whether the judge's confidence can be trusted. The authors ask whether a decision-only judge can serve as a cheap first pass, and also identify when stronger evaluation is needed. Their claim is that it can, and that its own confidence tells you when to escalate.

The judge is jev-as-a-judge. It was compared with sixteen generative and reward-model judges, with blinded human adjudication. On ordinary preference and on evidence-grounded factuality, it comes within three percentage points of a state-of-the-art LLM judge, the strongest comparator, at zero point three six percent of that comparator's fee.

Larger gaps appear when a judgment requires checking a derivation, or resisting a wrong answer that is elaborately written. On several benchmarks, JEV's gap to the comparator is concentrated in its low-confidence decisions. That is what makes a cascade possible. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains ninety-nine percent of the comparator's accuracy at lower cost.

The near-parity holds for ordinary preference and factuality. It does not hold for checking derivations or for resisting elaborate wrong answers. And the concentration of errors in low-confidence decisions is reported on several benchmarks, not all.

The title states the policy in six words: accept when confident, escalate when unsure. Where the cheap judge is sure, it is usually right, and where it is unsure, a stronger judge can take over.