Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders

Understanding On-Policy Distillation. A Mechanistic Interpretability Perspective via Sparse Crosscoders. By Zichao Yu and three others, from the University of Hong Kong, the University of Cambridge, and one other institution. Published on the twenty-eighth of September. No code repository is linked to it.

On-policy distillation, OPD, is a widely used post-training technique for reasoning in large language models. The common belief is that it transfers knowledge from a stronger teacher. What it actually puts into the student's internal representations has been unclear. The answer this study arrives at runs against the belief. OPD does not hand over new features. It reweights features the student already has. The student learns from the teacher how to use the features the two of them already share.

The tool is a sparse crosscoder, which learns one feature dictionary shared by three models: the student before OPD, the student after it, and the teacher. Standard crosscoder analyses can find features specific to one model, but they cannot show how a model's use of its features changes, because every model is encoded into one shared set of activations. The authors therefore propose the swap readout. It reads each student checkpoint's feature activations on their own, measuring how training shifts the use of each feature, even for checkpoints the crosscoder never saw.

Across three OPD settings, OPD neither creates features nor passes on the teacher's own. For over ninety-eight percent of the student's frequently used features, firing rates stay within twenty percent of where they started.

The study then turns to the SFT warm-up on the teacher's rollouts that commonly comes before OPD and makes it more effective. That warm-up, too, adds no features. It reweights the shared ones, in two ways. First, it already raises and lowers many of the features that OPD later raises and lowers, doing part of OPD's work in advance. Second, it changes features OPD alone would leave alone, notably those for conversation format, reasoning style and mathematical notation, and those changes persist through OPD.

The strongest test is causal. Take a student that was distilled directly, without the warm-up, and impose the warm-up's reweighting on its features without touching its weights. Its accuracy comes close to that of the warmed-up student. Apply the same change to shuffled features and it does not.

The results come from three OPD settings and from the lens of a sparse crosscoder, so the claim is about what that dictionary can see. The authors frame their conclusion as a suggestion that OPD reweights rather than acquires, built on those settings, the swap readout, and the intervention test. The statement about firing rates concerns frequently used features specifically.

Distillation is often described as pouring knowledge from a full vessel into an empty one. On this evidence the better picture is a dial board: the student and teacher share the same dials, and what passes between them is how far each one should be turned.