Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Rethinking On-Policy Distillation of Large Language Models II. One Training Example. By Zixuan Fu and twelve others, from Tsinghua University, the University of Chinese Academy of Sciences, and one other institution. Published on the third of September. Its code repository is worked out: about five hundred source files, with a readme and a licence.

On-policy distillation combines student-generated rollouts with dense token-level supervision from a teacher. The student produces the text, and the teacher supervises every token of it. Existing work has mainly studied the algorithmic behaviour of that arrangement, which leaves an obvious question unexamined: what is the training data actually contributing? This work answers it by going to the data-minimal limit and training on a single query. The result is the claim. One-shot on-policy distillation keeps improving for hundreds of steps, and recovers most of full-data training's gain, across task domains and across model families. One example. Most of the benefit.

The explanation is built on two quantities. The first is state coverage: the fraction of states that full-data training visits which a given query set's rollouts actually reach. A single query already reaches seventy-one point five percent of them, and most of that within the first one hundred steps. Adding semantically distinct queries raises coverage and validation accuracy together, until sixteen queries reach ninety-eight point nine percent and match full-data training outright. The second quantity is the rate at which the student aligns with the teacher, and this is where the asymmetry shows. Alignment slows at a similar pace whether training runs on one query or on the whole dataset. Even a fixed set of states takes hundreds of steps to absorb. The conclusion the authors draw from the pair is that on-policy distillation is data-overfed but algorithm-starved: rollouts expose broad supervision quickly, while the student absorbs that supervision increasingly slowly.

The state-coverage result extends beyond the single-teacher case. In multi-teacher distillation, sixteen semantically diverse queries per domain match full-data training there too, so the effect is not an artifact of having one teacher to align with. Then a stress test pushes considerably further. Two kinds of query that ought to be useless are tried: content-light templates, which carry almost no task material at all, and off-domain WildChat queries, which carry the wrong material. Both approach the real-query baseline. That is the point to hold carefully, because of what it implies rather than what it measures. Task content and induced state coverage can come apart. The queries do not have to be about the thing you are teaching, which means what a query set is contributing is the region of state space the student wanders into while answering it, not the knowledge the query appears to contain.

If this holds, the bottleneck in on-policy distillation was never the size of the dataset. It is the number of steps the student needs to absorb supervision it was already being shown, which points future work at step efficiency rather than at collecting more data.

Rethinking On-Policy Distillation of Large Language Models II. One Training Example. By Zixuan Fu and twelve others, from Tsinghua University, the University of Chinese Academy of Sciences, and one other institution. Published on the third of September. This is the long version: the whole paper rather than the abstract.

On-policy distillation has quietly become standard equipment in frontier post-training. Qwen3, MiMo, GLM-5, DeepSeek-V4 and Kimi K3 all use it alongside supervised fine-tuning and reinforcement learning. A growing literature examines how it works, almost entirely from the algorithmic side. What nobody had studied is how the training data shapes it. This paper closes that gap by taking the data away almost entirely, and the result is strange enough to need explaining.

First, what makes the method distinctive. The student samples its own rollouts, and at every prefix it visits, the teacher supplies the full next-token distribution. That combination is the point: on-policy state visitation, plus dense token-level supervision, rather than the single outcome-level reward that reinforcement learning with verifiable rewards provides. The experimental move here is borrowed from that neighbouring field, where someone ran one-shot reinforcement learning as an extreme controlled experiment to isolate the role of data. So: one-shot on-policy distillation. Train on a single query. To the authors' stated surprise, the learning curve looks strikingly like the one-shot reinforcement learning curve. The model keeps improving for hundreds of steps on one repeated example and ends up with a substantial gain. It holds across four task domains and three model families, and persists even on queries the student never manages to solve.

The data half is answered by asking what a query is actually worth. It acts on the student through the states its rollouts reach, where a state is a prefix at which the teacher supplies supervision. So a query set is worth the portion of that space it covers. The authors make this measurable as state coverage: group the states that full-data training visits into semantic clusters, then report what fraction a given setting's rollouts reach. One query reaches seventy-one point five percent of them, most of that within the first hundred steps. Sixteen semantically diverse queries reach ninety-eight point nine percent and match full-data training outright. That is why so little data improves the student so much. The algorithm half is answered by a different quantity, the absorption rate: the proportion of the remaining teacher-student gap that a single update closes. It falls as training proceeds, and it falls at essentially the same pace on one query as on all seventeen thousand. The pace of a run is a property of the optimizer, not of the training set.

Put those together and you get the paper's phrase. On-policy distillation is data-overfed but algorithm-starved. The rollouts expose broad supervision almost immediately, and then the student spends hundreds of steps absorbing an ever-smaller share of what remains.

The state-coverage result carries over to multi-teacher distillation, where sixteen semantically diverse queries per domain match full-data training there too. And the absorption-rate finding is what gives the paper its recommendation: the bottleneck was never the size of the dataset, so future work should target step efficiency instead. They suggest reusing a batch for several epochs under a trust region on the per-token gap, or weighting tokens by how much teacher signal they still carry.

The limitations are unusually well-matched to the claims. State coverage is a semantic-level proxy, measured against a reference space built from full-data rollouts, so it reports how much of that space a query set reaches rather than what it covers in absolute terms. It also weights every cluster equally, regardless of how often that cluster is visited or how much teacher signal it still carries. What actually sets the absorption rate remains open, which is a notable gap given that the paper's own conclusion points future work at it. And the multi-teacher runs use three domains with one teacher each, so how far the diversity result extends as teachers are added is untested.

The most useful shift here is in what a training set is for. If a query's value is the states it induces rather than the content it carries, then the question stops being how many problems to collect and becomes which states they lead the student into. The authors name that as the obvious next step, and also name what blocks it: state coverage currently needs a full-data run to measure against, so it cannot yet be used to select data before you have the data.