Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

Gains and Collapse in On-Policy Distillation. A Reinforcement Learning Perspective. By Han Cui and five others, from Zhejiang University and Westlake University. Published on the second of October. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.

On-policy distillation, or OPD, has become an important way to post-train language models. It brings performance gains, but it can also collapse into generation that is excessively long and repetitive, and why the same method goes one way or the other has been poorly understood. The authors explain both outcomes with a reinforcement learning reading. The teacher acts as an implicit reward model: it rewards behaviours of the student, including behaviours the teacher itself rarely shows.

Seen this way, the experiments say something precise about the gains. OPD improves performance without expanding what the student is capable of. When the implicit reward model is reliable, OPD makes the student's correct responses easier to sample. The collapse is the same mechanism pointed the wrong way. When the teacher's preference misaligns with quality, reward hacking happens. The implicit reward amplifies overlong, repetitive student rollouts, even though the teacher rarely generates text like that itself.

That is the uncomfortable part of the picture. The teacher does not need to write long, repetitive text for the student to be pushed toward it. It only needs to favour such text when the student produces it, and the training amplifies whatever is favoured. The diagnosis also suggests remedies, and the authors find two that each work on their own. Masking unhealthy responses during training mitigates the collapse effectively. So does starting from SFT initialization. Either one alone is enough to help.

The claim has a defined reach. It concerns how OPD reshapes what a student already does, by amplifying behaviours the teacher's implicit feedback favours. It does not say OPD gives the student new capabilities; the experiments say the opposite. And the two mitigations are offered as effective against the collapse, not as a full account of when OPD will succeed.

The practical shift is in what to ask of a teacher. The usual question is how well the teacher generates. This work moves the question to how reliably the teacher evaluates the student's rollouts, because under OPD that judgment, not the teacher's own output, decides what gets amplified.