Rethinking Relative Policy Differences Transfer of On-Policy Distillation

Rethinking Relative Policy Differences Transfer of On-Policy Distillation. By Rong Wu and eleven others, from Zhejiang University, the Shanghai Artificial Intelligence Laboratory, and three other institutions. Published on the ninth of October. Its code repository is worked out: about eight hundred and fifty source files, with a readme and a licence.

On-policy distillation trains a target policy on its own rollouts, with token-level supervision from a stronger teacher. It usually depends on that teacher being substantially more capable than the target. Recent work relaxed the requirement. Instead of transferring the expert's absolute distribution, it transfers the relative difference between an expert policy and a reference policy. What has stayed unclear is which differences actually carry useful supervision, and which targets can benefit from them. This work sets out to answer both, and its headline answer is that strength is the wrong thing to look for. A stronger expert is not necessarily a better source.

The answer rests on a unified view. The log-probability ratio between a proxy expert and a proxy reference is read as an implicit reward for KL-regularized policy optimization. That reading induces an on-policy update for the target. It recovers conventional on-policy distillation as a special case, and it allows any-to-any transfer between models that differ in scale, in training stage, or in generation.

Systematic experiments yield three findings. The first concerns where a useful difference comes from. Differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently. Differences that arise from model scale alone transfer weakly, or not at all. Put plainly, a larger model sitting next to a smaller one is not, by that fact alone, a useful teacher. A model that has been trained for the task, set against the version of itself before that training, is.

The second finding is that expert and reference play complementary roles. The expert supplies the capability to be transferred. The reference filters out behaviour the two share, isolating the residual that matters. Without a reference, the target would be pulled toward everything the expert does, including what it already does itself. With one, it is pulled only toward what is different.

The third finding is that even a useful difference does not reach every target. Transfer succeeds only when the induced update moves the target toward the expert and generalizes beyond the rollouts that were sampled. Both conditions concern the target, not the source. The same difference can help one model and do nothing for another.

Two further results bound how the findings should be used. Raw policy distance does not predict whether a transfer will work, so measuring how far apart two models are is not a shortcut. And the choice of reference is not incidental: an appropriate reference can rescue a transfer that would otherwise fail. The results are framed as characterizing when relative policy differences give effective supervision for on-policy distillation, which is the setting they speak to.

The picture that emerges replaces a single question with two. Rather than asking how strong the teacher is, one asks what training produced the difference between expert and reference, and whether the update it induces actually points this particular target toward the expert. Scale on its own answers neither, while task-specific training and a well-chosen reference answer both.