What Does Privileged Information Add to On-Policy Self-Distillation?

What Does Privileged Information Add to On-Policy Self-Distillation?. By XiuYu Zhang and four others, from the National University of Singapore. Published on the seventeenth of September. Its code repository is public and real: about forty source files.

On-policy self-distillation lets a language model learn from a frozen copy of itself that gets to see extra information: an answer, or a worked solution. Giving the teacher that privileged view seems to offer the student more to learn. The question is how much it adds beyond distillation itself. The answer here is that much of the gain comes from distillation without any reference at all. Much of what looks like learning from the answer is learning from the teacher itself, and the student would get it even without the answer in view.

To isolate the contribution, the authors build AMPLE-Math, a reusable suite of five thousand three hundred and nineteen math problems, each with six reasoning views that share the same answer. Each view is compared against matched distillation with no reference. Six views of one problem, all agreeing on the answer, let the study vary how much of the solution the teacher sees while holding everything else fixed. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for much of Qwen3-1.7B's improvement under thinking-enabled evaluation, both in domain and on external benchmarks.

Evidence for an extra benefit from the reference is modest in Qwen, and strongest for a polished solution. In SmolLM3-3B, complete traces add two percentage points at step fifty. The benefits also depend on the student being trained. At the same checkpoint, swapping short direct-response rollouts for long thinking-enabled ones turns gains into losses in both families, with the problems, references and evaluation all held fixed. Same references, same problems, same test, and the sign of the effect flips with nothing changed except how the student generates during training.

Teacher profiles and matched loss interventions in Qwen go further, showing that changing token-level supervision can leave the student's behaviour largely unchanged. The student is less sensitive to exactly what the teacher says, token by token, than the setup might suggest.

These are results for mathematics, in two model families, under particular rollout and evaluation modes, and the authors frame their conclusion as a suggestion. The effects move with how the student is sampled: the same reference that helps with short rollouts can hurt with long ones.

The proposed reading is that self-distillation works by improving access to reasoning abilities the model already has, through the parameters that direct-response and thinking-enabled inference share. On that view, a privileged reference is valued by what it adds to that transfer between modes, not by how much of the solution it gives away.