When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better. By Siyan Zhao and ten others, from NVIDIA and UCLA. Published on the eighth of October. No code repository is linked to it.

On-policy distillation, or OPD, has become an increasingly popular way to transfer a teacher's capabilities to a student model. The authors ask whether on-policy sampling is always beneficial, for any teacher-student pair, and their answer is no. A simple alternative they call Semi-OPD distills from offline rollouts generated by the initial student, and it can often outperform OPD in both accuracy and training efficiency. The difference between the two is where the rollouts come from. In OPD they are sampled from the student as it trains. In Semi-OPD they are produced up front by the student as it was at the start, and the distillation then runs on that fixed, offline set.

The comparison spans seventeen teacher-student pairs, ranging from one point five billion parameters up to two hundred and thirty-five billion. Semi-OPD outperforms OPD in fourteen of them. The accuracy gain reaches as much as thirteen point six percent, and the training speedup as much as eleven point four times. So in most of the pairs tested, the cheaper offline recipe was also the more accurate one.

What decides the winner is alignment between the initial teacher and the student, quantified by an output-token overlap ratio. OPD is beneficial only when the two are highly aligned, with high overlap. The deeper investigation suggests why. Effective distillation requires on-policyness with respect to both the student and the teacher. For a misaligned pair, student rollouts can become increasingly off-policy relative to the teacher as context length grows, and that weakens the distillation signal. Semi-OPD is often more stable. It distills on shorter contexts while still covering full trajectories, and it exposes the student to more teacher-preferred tokens.

The claim is deliberately conditional. It does not say on-policy distillation fails. It says on-policy sampling is not always the better choice, and the title's own word is often, not always. In the remaining three of the seventeen pairs, Semi-OPD did not come out ahead. The deciding variable is how aligned teacher and student are at the start, and when they are highly aligned, OPD is the one that benefits. The mechanism offered is also hedged: the deeper investigation suggests, rather than proves, that on-policyness toward the teacher is what misaligned pairs lose as their contexts lengthen.

Beyond proposing Semi-OPD as an efficient alternative, the authors want the community to rethink when to use OPD, and to study stronger OPD variants with meaningful teacher-student pairs. The practical upshot is a question to ask before picking a recipe. How much do this teacher and this student already agree on the tokens they produce?