On-Policy Distillation Teaches New Skills but Not New Knowledge

On-Policy Distillation Teaches New Skills but Not New Knowledge. By Yixuan Tang and one other, from the Hong Kong University of Science and Technology. Published on the seventh of October.

On-policy distillation strengthens reasoning in language models. What has been unknown is what the student actually picks up from the teacher: new factual knowledge, or the compositional skill of chaining steps together. The claim is that it picks up the skill and not the facts. Distillation of this kind does not expand a model's parametric knowledge. It teaches the model to organize and compose the knowledge it already has.

To pull the two apart, the authors use a controlled synthetic framework. It measures what the student can do at the start, and independently controls whether the teacher holds extra facts, extra compositional skill, or both. Across four models from three families, reverse-KL on-policy distillation reliably transfers compositional skill, even to reasoning structures not seen before, yet transfers minimal factual knowledge.

Taking the recipe apart shows where the asymmetry comes from. Swapping reverse KL for forward KL restores the transfer of facts, while the student's own rollouts are what specifically improve the execution of multi-step reasoning. Experiments on recent factual question answering and on competition mathematics show a similar pattern: notable reasoning gains, with no expansion of factual memory.

The claim attaches to a particular recipe. It is reverse-KL on-policy distillation that fails to move facts, and the forward-KL variant does move them, so the limit belongs to the divergence, not to distillation as such. The cleanest evidence comes from a synthetic setting, with four models from three families, backed by the factual and mathematical experiments.

The practical reading is a division of labour inside one recipe. The divergence decides whether facts come across, and the student's own rollouts decide how well it strings steps together, so a gain in reasoning should not be mistaken for a gain in what the model knows.