One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

One Symptom, Three Levers. A Critical Review of On-Policy Self-Distillation. By Justin Robert and one other, from O V H A I L L M. Published on the twenty-sixth of August. No code repository is linked to it.

On-policy distillation trains a language model on text it generated itself, while a teacher scores that text token by token. The appeal is a combination. From imitation learning it takes dense supervision, a signal at every token rather than one at the end. From reinforcement learning it takes on-policy sampling, so the model learns on its own behaviour and not on someone else's. The price is a second model. A larger one has to be kept around to act as teacher.

On-Policy Self-Distillation, or OPSD, removes that price. The teacher becomes the model itself, but conditioned on privileged information the student will not have at test time: a reference solution, say, or a plan, or feedback from the environment. The teacher is no stronger than the student. It is only better informed. Early results were promising, reaching accuracy comparable to reinforcement learning while generating only a fraction of the tokens.

The review is built around one failure mode, because the authors say it now dominates the field. That failure is collapse: the progressive narrowing of the set of reasoning paths the model is able to produce. Its source is the same asymmetry that makes the method work. The gap in information between teacher and student is what produces the signal, and that gap is also what biases it. Collapse is not unique to OPSD, the review notes, though privileged information makes it worse.

Rather than cataloguing papers, the review treats collapse as a symptom and sorts the field by three levers that govern it. The first lever is where the signal is applied, meaning how the individual tokens are weighted. The second is what the teacher is shown, meaning the nature of the privileged information it is conditioned on. The third is when the signal changes, meaning how the teacher itself moves over training and how its guidance decays.

The scope is deliberately narrow. The review stays inside mathematical reasoning, which is where the method originated and where its failure modes are best documented. It reports no new experiments of its own.

What it offers instead is structural, and it comes in two parts. One is a shared vocabulary, since the same phenomena have been given different names in different papers, and a reader following the literature can easily miss that two results are describing one thing. The other is a line drawn clearly between what is settled and what is still disputed. For anyone trying to read the area, that line separates claims that can be built on from those still being argued over.

The title carries the whole argument in compressed form. One symptom, which is collapse. Three levers, which are token weighting, the teacher's privileged information, and the teacher's dynamics over time. A self-distilling model learns from a version of itself that knows a little more, and the review's contention is that how much it knows, where its signal lands and how that signal fades are the settings that decide whether the student keeps its range of reasoning or loses it.