On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics. By Julianna Piskorz and two others, from the University of Cambridge. Published on the twenty-eighth of September. No code repository is linked to it.

On-policy learning has been credited with three virtues: less catastrophic forgetting, sparser parameter updates, and better generalisation. But the comparisons behind those claims, between supervised fine-tuning and reinforcement learning, change many factors at once, which makes it hard to tell what the rollout policy itself contributes. Rollout policy here means whose outputs the student trains on: its own generations, which is the on-policy case, or the teacher's, which is the off-policy one. This study isolates that single choice, and the picture it finds is nuanced. The rollout policy does not necessarily play a central role. Two other knobs matter more. The direction of the token-level KL more clearly shapes task performance and output coverage. The learning rate governs forgetting and update sparsity. In other words, two of the virtues credited to on-policy learning, less forgetting and sparser updates, track the learning rate instead.

The setting is a controlled distillation from a strong teacher to a weak student. Three factors are varied independently: rollout policy, KL direction at the token level, and learning rate. The experiments cover the Llama3 and Qwen2.5 model families, on reasoning tasks drawn from scientific, medical and arithmetic domains.

To explain the pattern, the authors analyse KL gradients and run experiments along a continuous spectrum of rollout policies, from fully the student's to fully the teacher's. Forward KL proves remarkably robust to rollout policy: its performance stays stable and strong whichever rollouts it sees. Reverse KL is substantially more sensitive, and it favours rollouts the student generates itself. So how much the rollout source matters depends on which way the KL points, and with forward KL it barely matters at all. The broader conclusions survive several stress tests: removing gradient clipping, using sampled KL estimators, and training on tasks that need longer chains of reasoning.

On-policy data is not left without a role. It does improve generalisation to harder variants of the Countdown arithmetic task, under both KL directions. That advantage, however, does not reliably persist after a subsequent stage of RLVR. And the whole study lives inside one regime, strong-to-weak distillation, with two model families and reasoning tasks in three domains, so its verdict is about that setting rather than about reinforcement learning at large.

The result reframes a familiar debate. Instead of asking whether on-policy rollouts are better, the question becomes better under which objective, judged how, and at what learning rate. The authors' answer is that the value of on-policy data depends critically on all three, and is not a property it carries on its own.