Judging the Justification: Decision Certificate Supervision for Generative Reward Models

Judging the Justification. Decision Certificate Supervision for Generative Reward Models. By Guiquan Sun and three others, from the University of Connecticut and NEC Labs America. Published on the twentieth of October. Its code repository is public and real: about thirty-five source files.

Generative reward models, or GenRMs, do not just pick a winner between two responses. They write a natural language rationale justifying the preference. A correct verdict, though, does not guarantee a sound justification. The usual way to supervise rationales leans on reference explanations, and that has two drawbacks. References can be costly to obtain, and having one does not necessarily establish whether a generated rationale actually supports the verdict it arrives at. A judge can land on the right answer for the wrong reasons, and training that only checks the answer will never notice.

DeCert evaluates generated rationales without any reference explanation. It asks two things of a rationale: whether it is valid, and whether it is consistent with the verdict. Those assessments are then turned into preference supervision, so the training signal reflects both whether the verdict is correct and how good the justification is. The name comes from treating each rationale as a decision certificate: a piece of writing that is supposed to establish the verdict, and can be checked for whether it does.

The useful case is when several sampled judgments agree, and agree on the correct verdict. Verdict correctness alone cannot separate them. Their certificates can. Differences in certificate quality let DeCert construct rationale preference pairs, pairs that distinguish a stronger justification from a weaker one even though both reached the right answer. That turns agreement, which would otherwise carry no signal, into something to learn from.

The empirical studies span two model families and several benchmarks, both in distribution and out of distribution. Across them, DeCert consistently outperforms strong generative judge baselines, and rationale quality improves at the same time. One specific failure is singled out: positional bias. DeCert mitigates it, raising position-consistent accuracy by as much as fourteen point eight points.

For people training reward models, the open question this paper presses is what a judge is rewarded for. The authors' answer is that supervision should cover not only which verdict is correct but how that verdict is justified, and they do it without paying for reference explanations.