Training LLM Judges from Language Feedback via Position-Selective Self-Distillation

Training LLM Judges from Language Feedback via Position-Selective Self-Distillation. By Ilgee Hong and seven others, from the Georgia Institute of Technology, Amazon, and one other institution. Published on the thirtieth of September. Its code repository is worked out: about eight hundred and fifty source files, with a readme and a licence.

This work is about training language-model judges from natural language feedback, with a focus on subjective tasks. On those tasks, the verdict depends heavily on which evaluation criteria the judge brings up and how it weighs them. The dominant training approach is outcome-supervised reinforcement learning, GRPO for instance. It gives every token in a rollout the same single scalar, decided only by whether the final verdict was accurate. Two things are lost that way. The tokens where the judge chooses its criteria get no separate credit. And the rich language feedback that naturally comes with preference labels, such as the rationales behind a preference, is ignored entirely. A judge can reach the right verdict for the wrong reasons, or the wrong verdict after choosing sensible criteria, and a single scalar on the outcome cannot tell those cases apart at the tokens where the choice was made.

Self-distillation is one natural way to put that feedback to use. The same model, conditioned on the feedback, acts as a teacher and provides dense supervision at every position. The paper's point is that not every position deserves the same treatment, because they do not all carry equally useful signal.

To sort them, the authors measure the entropy shift between teacher and student at each position, and find two regimes. In context sharpening, the teacher concentrates its probability on one particular way of expressing a criterion that agrees with the feedback. In context spreading, the teacher spreads probability across several alternatives that all agree with the feedback. Their reading is that sharpening encourages memorizing one specific expression of a criterion, while spreading encourages semantic understanding, because it keeps the alternatives alive.

That asymmetry motivates a mask. Positions are masked according to their entropy shift, and only the lower tail of the entropy-shift distribution is kept for distillation. Positions with a higher entropy shift are simply left out of the teacher's supervision.

Masking those higher-entropy-shift positions improves out-of-distribution generalization compared with naive self-distillation. The judges trained this way beat judges trained with outcome-supervised reinforcement learning by two to nine percentage points on the subjective subcategories evaluated, and stay competitive on the objective ones. The pattern fits the paper's framing. The gains appear where the verdict hinges on which criteria are invoked and how they are weighed, which is exactly where outcome-only credit had nothing to say about the criterion-choice tokens.

Picture two signals passing through the same training run. One is a single grade on the final verdict, spread evenly over every token. The other is the rationale behind a preference, delivered position by position by the model itself once it has read that rationale. The method keeps the second and then throws part of it away on purpose, and the evidence says that discarding the right part is what lets the judge generalize.