Order-Only On-Policy Distillation

Order-Only On-Policy Distillation. By Guangzeng Han and two others, from the University of Memphis and the University of Missouri. Published on the first of November.

On-policy distillation, or OPD, trains a student on its own generations by matching the teacher's token probabilities. Those probabilities carry two kinds of information tangled together. One is which tokens the teacher prefers. The other is how confident it is. That second part reflects the teacher's own capacity and calibration, which a smaller student may simply be unable to reproduce. There is a practical symptom too. The teacher temperature changes the probability values without changing their ordering, so OPD ends up sensitive to whichever temperature is chosen.

The question posed is whether the ordering alone is enough. Order-Only On-Policy Distillation keeps the teacher's preferences and lets go of its confidence. Rather than fitting how much probability the teacher puts on each token, the student learns only which tokens the teacher ranks above which. Put plainly, the student is asked to agree with the teacher about what comes first, second, and third, and is no longer asked to be exactly as sure as the teacher is. Confidence that the student may not be able to reproduce simply drops out of the target.

At each prefix the student generates, the method trains the student to reproduce the order of the teacher's top-K tokens, without fitting their probability values. One consequence follows directly. Since temperature reshapes the values but leaves the order intact, the objective is invariant to the teacher temperature, and there is no temperature left to tune. The supervision stays on-policy: it is applied to the student's own generated prefixes, as in OPD.

The comparison spans six teacher-student pairs, scored on the average over HMMT25, AIME24, and AIME25. Order-only distillation beats OPD on that average in every pair, by up to two point nine points, with gains of up to five point six points on AIME25 alone. Swapping in other ranking losses yields similar gains, which places the benefit in ordinal supervision itself rather than in one particular loss.

The analysis adds a twist. The order-only student departs further from the teacher's probabilities than an OPD student does, yet it reproduces the teacher's ranking more faithfully, and it ends up the stronger student. By the measure OPD optimises, it looks like the worse copy. By the measure the method optimises, and by the benchmark results, it is the better one.

That combination is the argument in miniature. Matching the teacher's numbers more closely did not make the better student; matching its preferences did. The ranking, not the probability values, carries the knowledge that on-policy distillation needs to transfer.