On the Off-Policy Teacher in On-Policy Distillation
Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Blöbaum, Purak Jain
Washington University in St. Louis · AWS AI Labs · Carnegie Mellon University · Georgia Institute of Technology
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 21 · arXiv 2609.38360
The abstract
On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by the student. Empirically, we find that its continuation performance degrades as these prefixes grow longer. To address this issue, we propose Student-COnditioned Updates of the Teacher (SCOUT), a co-training framework that adapts the teacher to student-generated prefixes. Alongside standard OPD updates, SCOUT periodically optimizes the teacher's conditional ability using reinforcement learning with verifiable rewards, where the teacher generates continuations from student prefixes and learns from outcome rewards. Controlled experiments show that SCOUT improves the teacher's ability to continue from student-generated prefixes, supporting the intended mechanism of student-conditioned teacher adaptation. Across multiple teacher--student configurations, model scales, and reasoning domains, SCOUT also consistently improves the effectiveness of on-policy distillation.
The Brief
2:22A short podcast summary of the paper. Written and voiced by AI.
Transcript
On the Off-Policy Teacher in On-Policy Distillation. By Langlin Huang and seven others, from Washington University in Saint Louis, AWS AI Labs, and two other institutions. Published on the twenty-ninth of September. No code repository is linked to it.
On-policy distillation, OPD, is a post-training approach that has recently gained ground. The student generates trajectories from its own policy, and a teacher supervises them densely. There is an asymmetry built into that arrangement. The trajectories are on-policy for the student, but they are off-policy for the teacher. A teacher is typically optimised to continue from prefixes it generated itself. In OPD, it must instead supervise prefixes the student wrote. The authors find empirically that the teacher's continuation performance degrades as those student prefixes grow longer. Put another way, the further into its own attempt the student gets, the less dependable the teacher becomes as a guide to the next step. The dense supervision OPD promises is thinnest where the student has travelled furthest from anything the teacher itself would have written.
If the teacher is out of its element on student text, the remedy proposed is to train the teacher on student text. SCOUT, short for Student-COnditioned Updates of the Teacher, is a co-training framework that adapts the teacher to prefixes generated by the student, while distillation carries on.
Standard OPD updates continue as usual. Alongside them, SCOUT periodically works on the teacher's conditional ability, using reinforcement learning with verifiable rewards. In those updates, the teacher is handed prefixes produced by the student, generates continuations from them, and learns from the outcome rewards those continuations earn. The teacher is, in effect, practising the exact task OPD asks of it.
Controlled experiments check the mechanism directly. SCOUT does improve the teacher's ability to continue from prefixes generated by the student, which is what student-conditioned adaptation was meant to achieve. The downstream effect follows: across multiple teacher and student configurations, multiple model scales, and several reasoning domains, SCOUT consistently makes on-policy distillation more effective.
The title's question gets a plain answer. In on-policy distillation the teacher is the one working off-policy, continuing someone else's half-finished work, and it weakens as that work gets longer. SCOUT's response is to let the teacher keep learning alongside the student, on exactly that kind of input.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a teacher out of its depth on the student's own words.
You are post-training a small student model. It writes its own attempts at maths problems, and a stronger teacher scores every token of them, giving dense guidance on the student's own behaviour.
The road ahead
- 1 · The problem ~7 min A strong teacher grades every token of a student's own maths attempts, but it gets worse at continuing those attempts the longer they run.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Train the teacher itself on student-written prefixes, rewarding correct final answers, while the student keeps learning from it token by token.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Sample student attempts, cut prefixes from them, let the teacher continue those prefixes, and update the teacher with outcome rewards.
- 6 · The experiment ~6 min You’ll guess: Across teacher and student configurations, model scales and reasoning domains, how did SCOUT fare against standard on-policy distillation?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.