Learning from Teacher Continuations at Student States

Learning from Teacher Continuations at Student States. By Haojin Wang and ten others, from the University of Illinois at Urbana-Champaign, the Massachusetts Institute of Technology, and three other institutions. Published on the twenty-eighth of September. No code repository is linked to it.

Existing ways of distilling a teacher language model into a student each carry a known weakness. Offline supervised fine-tuning on fixed teacher trajectories suffers sequential covariate shift: the student trains on the teacher's paths but must then generate from its own. Token-level on-policy distillation gets fragmented supervision when a prefix has already gone wrong. And distribution-matching distillation needs access to the teacher's token probabilities, which is not always available.

OLIVE, short for OnLine InterVEntion, is built so that each of its design choices answers one of those weaknesses. At every iteration, the evolving student generates a new prefix. The teacher continues that prefix autoregressively. The student is then updated with cross-entropy on the tokens the teacher generated.

Read against the three weaknesses, the design lines up neatly. Because prefixes come from the current student, training happens at the states the student actually reaches. Because the teacher writes a whole continuation, supervision is not fragmented when a prefix goes astray. And because the loss is computed on the teacher's text, the teacher's token probabilities are never needed. An asynchronous implementation cuts OLIVE's total training time by a further twenty-three point eight percent.

OLIVE reaches higher reasoning performance than on-policy distillation with a top-sixteen KL approximation, at comparable cost in GPU hours. On hard reasoning tasks and on agentic tasks, which reflect modern post-training settings, it consistently beats existing distillation methods under the same training budget. Because prefixes keep being regenerated from the evolving student, OLIVE keeps improving after offline distillation plateaus, and it better preserves the student's general capabilities and plasticity. Using only text from GPT-5.4-mini, continued training with OLIVE beats offline supervised fine-tuning from the same teacher by thirteen percent on ScienceWorld.

The whole method is one small reversal. Instead of the student imitating where the teacher went, the teacher shows the way forward from wherever the student currently stands. Each of the three older weaknesses is met by a piece of that arrangement: the states come from the student, the continuation is whole, and the signal is plain text. That last point means the teacher can be a model seen only through its outputs, and the ScienceWorld result shows that working with a teacher known only by its text.