On-Policy Distillation with Negative-Policy Rollouts

On-Policy Distillation with Negative-Policy Rollouts. By Jaehui Hwang and three others, from NAVER AI Lab. Published on the sixth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

On-policy distillation, or OPD, is a widely studied post-training method. A student model generates its own rollouts, and a stronger teacher supplies token-level supervision on them. Recent work has refined it with alternative distillation reward formulations and different teacher configurations, but the objective has stayed the same: mimic the teacher. That positive guidance has a weak point. When the stronger teacher has limited distributional overlap with the student, imitation alone can provide insufficient learning signal.

Negative-Policy OPD, or NP-OPD, adds a second reference pointing the other way. Alongside the teacher, it brings in a negative policy, one that is lower performing and lower in capability, to serve as an example of what the student should move away from. Teacher supervision is complemented rather than replaced. The student still learns what to do from the teacher, and now also has a concrete picture of what not to do. Where the gap between teacher and student is too wide for imitation to say much, a reference from below offers a second direction to learn from.

Where the negative policy enters is the distinctive choice. NP-OPD does not modify the distillation reward formulation. It introduces the negative policy at the rollout stage. Throughout training, this continuously supplies tokens that the negative policy prefers over the teacher, so those tokens remain exposed to teacher supervision. The result is an explicit negative signal, arriving through the negative policy's rollouts, while the positive teacher supervision of ordinary OPD is kept intact. Put plainly, the weaker policy decides which tokens get placed in front of the teacher. Tokens the weaker policy favours, and the teacher does not, keep turning up during training, and each time they do the teacher's supervision applies to them. Nothing about how that supervision is scored has been altered. Only the material it is applied to has changed.

Through extensive experiments, the authors report that NP-OPD improves OPD across model scales, across generation modes, across reasoning domains, and across different OPD variants. The abstract states these gains in general terms and does not give figures.

Analyses then check that the method does what it was designed to do. NP-OPD effectively suppresses the tokens preferred by the negative policy over the teacher, and it moves the student away from the negative policy. The authors take these results as support for introducing negative signals through rollouts rather than through the reward. Code will be made available.

The broader point concerns a part of the method that usually goes unexamined. Most attempts to improve on-policy distillation adjust what the teacher says or how its supervision is scored. This one changes which tokens the teacher is asked to judge in the first place, by letting a weaker policy choose some of them. The authors frame the result as new insight into the role the rollout policy plays in OPD.