Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Md. Ismail Hossain, Humaira Kousar, Isidora Chara Tourni
North South University · KAIST · Andria Labs
Published September 29, 2026 · on the feed October 1, 2026 · ▲ 7 · arXiv 2609.37915
The abstract
On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solution. We separate these roles in a factorial analysis and find that scaffold correctness has a stronger effect on downstream accuracy than context correctness. Unverified scaffolds create an imitation gap because the teacher can use information unavailable to the student. This gap shrinks with model scale, yet OPSD continues to supervise mostly unverified trajectories. In contrast, verified scaffolds remain effective even when the teacher is conditioned on the student's own unsuccessful rollout. Based on this finding, we introduce OASIS, which retains the OPSD objective but supervises mostly verified by label on-policy trajectories and replaces written solutions with unverified model-generated attempts as the teacher context. OASIS therefore requires only final-answer labels. Across Qwen3-1.7B, 4B, and 8B on AIME 2024, AIME 2025, and HMMT 2025, OASIS improves over the base model by 3.2--3.8 points on average, while OPSD's gain falls from 3.05 points at 1.7B to 0.14 at 8B. At 8B, OASIS improves over OPSD by 3.05 points, showing that verified on-policy scaffolds preserve the effectiveness of self-distillation as models scale.
The Brief
2:38A short podcast summary of the paper. Written and voiced by AI.
Transcript
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning. By Md. Ismail Hossain and two others, from North South University, KAIST, and one other institution. Published on the twenty-ninth of September. No code repository is linked to it.
On-policy self-distillation, OPSD for short, trains a student to match a privileged teacher's distribution along the student's own sampled trajectory. In its standard form, two things happen at once. The supervision is applied to student rollouts that nobody has verified, and the teacher is conditioned on privileged context, typically a reference solution. The trouble shows up as models grow. OPSD's gain over the base model falls from about three points at the smallest scale tested to almost nothing at the largest.
The authors pull those two roles apart in a factorial analysis: whether the scaffold being supervised is correct, and whether the teacher's context is correct. Scaffold correctness turns out to matter more for downstream accuracy than context correctness. Their explanation is an imitation gap. When the scaffold is unverified, the teacher can lean on information the student does not have. That gap shrinks as models scale, and yet OPSD keeps supervising mostly unverified trajectories. Verified scaffolds, by contrast, stay effective even when the teacher is conditioned on the student's own failed rollout.
That finding becomes OASIS. It keeps the OPSD objective exactly as it was, and changes what it is applied to. Supervision goes mostly to on-policy trajectories verified by their label. And the teacher's context is no longer a written solution: it is an unverified attempt generated by a model. Put together, the method needs only final-answer labels. No reference solutions have to be written at all.
Experiments use Qwen3 models at one point seven billion, four billion and eight billion parameters, on two editions of AIME and on HMMT. Averaged, OASIS improves over the base model by between three point two and three point eight points. OPSD's gain, meanwhile, falls from three point zero five points at the smallest size to zero point one four at eight billion.
At that largest size, OASIS beats OPSD by three point zero five points. The pattern is the one the analysis predicted: the method that keeps supervising unverified scaffolds fades with scale, and the one that supervises verified scaffolds holds its gain.
Read in one line, the result relocates where correctness has to live. The teacher's context can be an unverified attempt, even a failed one. The trajectory being supervised is what needs to be right, and getting that right only takes a final answer to check against. That is what lets self-distillation keep paying off as the models get bigger.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a training trick that fades just as models get big.
You are training a maths model by self-distillation: the same model acts as a teacher that also sees a written reference solution, and the student learns to match its token choices along its own sampled attempts.
The road ahead
- 1 · The problem ~7 min Self-distillation with a reference-reading teacher helps small maths models, but the gain shrinks as models grow, while most supervised attempts fail and reference solutions cost effort.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Supervise the student along its own attempts that the final-answer label verifies, and let the teacher read another model attempt instead of a written solution.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Sample the student's attempts, check their final answers, give the teacher another attempt as context, and apply the unchanged OPSD loss along a verified attempt.
- 6 · The experiment ~6 min You’ll guess: Standard on-policy self-distillation (OPSD) helps small Qwen3 models. At the 8B size, how many points did OPSD gain over the base model?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.