Scaling Properties of Same-Family On-Policy Distillation
Yuntai Bao, Qinfeng Li, Guoqing Jiang, Liwei Chen, Zhiheng Qin, Xuanping Li, Wenqi Zhang, Xuhong Zhang
Zhejiang University · Kuaishou Technology
Published September 26, 2026 · on the feed September 30, 2026 · ▲ 315 · arXiv 2609.32722
The abstract
*Reinforcement learning (RL)* can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of *on-policy distillation (OPD)* across *weak-to-strong*, *same-base*, and *strong-to-weak* teacher--student setups. We find that early OPD training dynamics uniformly exhibit a regular *useful-transfer* regime, in which held-out accuracy (the *gold score*, G) rises approximately linearly in d=mathrm{KL(π_θVert π_{ref})}, the square root of token-level reverse KL divergence from the student initialization. In every observed weak-to-strong pair, the student's peak gold score exceeds its teacher's own, so a compact RL expert can transfer capability to a much larger student via OPD. To estimate OPD outcomes, we fit *power laws* for how G_{peak} and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws show that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score smaller teachers transfer better, so a teacher's score alone does not define its supervision value. We also study the scaling effects of two OPD variants, bootstrapping weak-to-strong OPD, and the degree of on-policy supervision.
The Brief
2:24A short podcast summary of the paper. Written and voiced by AI.
Transcript
Scaling Properties of Same-Family On-Policy Distillation. By Yuntai Bao and seven others, from Zhejiang University and Kuaishou Technology. Published on the twenty-sixth of September. No code repository is linked to it.
Reinforcement learning can give large language models substantial reasoning ability. How much of that ability carries over between models of different sizes, and how quickly, is less clear. This study examines on-policy distillation across three kinds of teacher and student setup: weak-to-strong, same-base, and strong-to-weak. The central finding is a regularity. Early in training, every setup shows what the authors call a useful-transfer regime. In it, held-out accuracy, which they call the gold score, rises roughly linearly in one particular distance: the square root of the token-level reverse KL divergence from where the student started. Put another way, the further the student moves from its own starting point, measured this way, the more it gains, and at first it gains at a steady rate.
A second finding concerns direction. In every weak-to-strong pair observed, the student's peak gold score ended up above the teacher's own. A compact expert trained with reinforcement learning can therefore pass capability on to a much larger student through on-policy distillation. To predict outcomes, the authors fit power laws for how the peak gold score, and the slope of the useful-transfer regime, scale with the parameter counts of student and teacher, and with the teacher's gold score.
Two conclusions come out of those laws. Peak gold score improves with teacher scale only up to roughly the student's own scale. And at a matched gold score, smaller teachers transfer better. A teacher's score on its own, then, does not define how much its supervision is worth.
The setting is distillation within one model family, as the title says, and the linear regime describes the early dynamics of training rather than a whole run. The power laws are fits to the setups studied. Within that frame, the study reaches in three further directions: the scaling effects of two variants of on-policy distillation, bootstrapping weak-to-strong distillation, and the degree to which the supervision is on-policy.
A natural guess would be that the strongest teacher available is always the best one. These results replace that guess with something more particular. Beyond the student's own size, a bigger teacher stops raising the peak. And between two teachers with the same score, the smaller is the better teacher. Choosing a teacher, on this evidence, means weighing its size against the student's, not just reading off its benchmark score.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
One small expert, a whole family of models to teach. The trouble is knowing what will happen before you pay for it.
You have a small model that reinforcement learning has made good at maths. You want its skill in bigger and smaller models of the same family, without repeating the costly training at every size.
The road ahead
- 1 · The problem ~6 min A small model trained hard on maths holds a skill you want in every model size, but nobody can say in advance how well passing it on will work.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Measure progress as the square root of reverse KL from the start, find accuracy rises in a line early on, and fit power laws that predict it.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Train many teacher and student pairs the same way, plot each run on one KL ruler, then fit laws across their sizes.
- 6 · The experiment ~6 min You’ll guess: When a small RL-trained teacher distills on-policy into a much larger student from the same family, how does the student's peak held-out accuracy compare with the teacher's?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.