On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation

On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation. By Yichen Wang and six others, from the University of Chicago, Duke University, and three other institutions. Published on the third of November.

On-policy distillation, or OPD, is used to pull capabilities out of a teacher model and into a student. Its multi-teacher extension, MOPD, consolidates capabilities from several teachers at once. Prior work reports near-perfect in-distribution distillation performance. The claim here is that such a number can hide something. Scoring well on the distribution that was supervised does not mean the capability was fully consolidated, and beyond that distribution the student can fall short of its teachers, and even of the model it started from.

A preliminary study on synthetic reasoning shows OPD not fully transferring the teachers' ability to generalize. The investigation then moves to real tasks, using Qwen3 models across mathematical reasoning, coding, agent work and long context, and testing two kinds of generalization: extrapolative and compositional.

In the extrapolative settings, OPD and MOPD reach between ninety-six point zero four and one hundred and two point seven three percent of their teachers' in-distribution scores. Yet their out-of-distribution scores stay below the base model's, by between two point four five and sixteen point two five percent. In three compositional settings, MOPD does improve in-distribution performance over the base model. On the composed tasks, treated as out of distribution, it remains between eighteen point one nine and twenty-four point three four percent below the best teachers.

Two possible bottlenecks are examined: the capability of the teacher, and the coverage of the data. Easing both together eliminates the out-of-distribution forgetting in these experiments, and points to the possibility of generalizing beyond the teacher. Relative to its reinforcement learning baselines, the distillation family also forgets less out of distribution on average. Its updates are smaller and more spectrally concentrated on average than the reinforcement learning updates, and MOPD shows more shared update structure. From these findings the authors propose geometry-based patching and base-preserving replay, which improve out-of-distribution performance while largely preserving in-distribution performance.

The result is bounded in ways the abstract keeps visible. The gaps are measured on Qwen3 models, in the settings listed. Removing the forgetting required alleviating both bottlenecks jointly, and was shown in the authors' experiments, while going past the teacher is described only as a potential. Nor is distillation cast as worse than reinforcement learning on forgetting: on average it does better. The blind spot concerns what in-distribution scores fail to reveal.

A student can score as well as its teachers on what it was trained on and still lose ground everywhere else. The authors' conclusion is that consolidation through distillation needs algorithms that explicitly support retention and generalization out of distribution, not only success within it.