On-Policy Self-Distillation with Teacher Evolution

On-Policy Self-Distillation with Teacher Evolution. By Shufan Shen and five others, from the Institute of Computing Technology, Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and one other institution. Published on the twenty-eighth of October.

On-Policy Self-Distillation, or OPSD, has become a promising way to ease reward sparsity in post-training that rewards whole sequences. A student model is aligned, token by token, with the distribution of its own self-teacher: the same model, prompted with privileged context. Yet OPSD often fails to keep improving as training goes on. The authors trace this to the teacher. The model's self-teaching capability is limited, and on certain samples it produces guidance that cannot be relied on. A student that faithfully copies an unreliable teacher on those samples has little reason to keep getting better.

If the teacher is the weak link, the teacher should get better too. That is the premise of Teacher-Evolution Self-Distillation, or TESD. The self-teacher generates its own rollouts and receives feedback from the environment. The rewards reflect two things: whether the task was done correctly, and context independence.

Those rewards are put to two uses. First, they serve as training objectives for teacher-oriented optimization, which keeps strengthening the model's self-teaching capability while distillation is under way. Second, they are used to estimate how reliable the teacher's guidance is on each sample. The student-oriented optimization is then adaptive, giving less weight to samples where the guidance looks unreliable.

Extensive experiments cover multiple datasets and several model scales. Across them, TESD consistently outperforms representative methods of both kinds: token-level methods, the family OPSD itself belongs to, and sequence-level methods, the post-training paradigms whose sparse rewards OPSD set out to ease. The abstract reports this as a consistent pattern rather than giving individual figures.

The paper locates the stall in self-distillation not in the student but in the quality of what it is being taught. TESD responds on both sides at once: it trains the teacher to teach better, and it tells the student when to listen less. Both moves are driven by the same rewards, so the signal that improves the teacher also measures how far to trust it. The authors read the results as evidence that improving supervision quality is necessary for OPSD to deliver steady gains.

On-Policy Self-Distillation with Teacher Evolution. By Shufan Shen and five others, from the Institute of Computing Technology, Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and one other institution. Published on the twenty-eighth of October. This is the long version: the whole paper rather than the abstract.

Reinforcement learning with verifiable rewards gives a model one score for a whole response, so it cannot tell which tokens actually mattered. On-policy self-distillation offers dense, token-level supervision instead, by turning the model into its own teacher. The trouble, the authors report, is that it stalls: its gains stop coming, and on their science benchmarks it finishes behind plain sequence-level training. Their diagnosis is that the teacher is the weak link, and nothing in the method ever makes it better.

Here is how the self-teacher works. The student answers a question from the question alone. The same model is then shown the question plus a privileged context, usually a correct solution taken from one of its own successful attempts, and asked how likely it finds each token of the student's answer. The loss is a per-token divergence between the student's distribution and the teacher's, with the teacher side held fixed. The teacher's weights trail the student's as an exponential moving average.

The authors first test whether that teacher can teach. On the Physics subset of SciKnowEval, they compare its confidence in each student answer with whether the answer was right. It gives low confidence to some correct answers and fairly high confidence to many wrong ones. And it shows a confidence-resemblance bias: answers that look like the privileged solution, by edit distance, get high confidence, while correct answers worded differently get low confidence. Tracked over training, their measure of teaching ability, a rank correlation between confidence and reward, drifts downward and fluctuates sharply.

The proposed method, Teacher-Evolution Self-Distillation, or TESD, lets the teacher act. The budget of sixteen rollouts per question is split: eight come from the student, and eight from the model prompted as teacher, with the privileged context in view. Each teacher rollout is scored by the environment on two things. The first is task correctness, one or zero. The second is independence: how far the rollout sits from the privileged context, measured as a normalized edit distance over lowercased words. A balancing weight on that second term is fixed at one throughout. The point of rewarding distance is to simulate the teacher's real job, which is guiding student answers that do not look like the reference. Those rewards are turned into group-relative advantages and the teacher is trained with a clipped policy-gradient objective, exactly as GRPO trains a student.

The same rewards do a second job. For each question, the mean reward of its teacher rollouts estimates how reliable the teacher's guidance on that question is likely to be. Rather than dropping weak questions outright, they pass these scores through a softmax across the batch, multiply by the batch size so the overall loss scale is kept, and clip the result. Each question's distillation loss is scaled by its weight, and the full objective adds the teacher's policy loss. For stability, the teacher's moving-average rate is cut from five hundredths to one hundredth.

The models are Qwen three at three sizes and Olmo three, trained on four science subsets of SciKnowEval, a ToolAlpaca tool-use task, and coding problems from LiveCodeBench. Every method gets the same sixteen rollouts. On the science and tool-use table, TESD has the best mean for all three models: seventy-four point three two percent on the four-billion model against seventy-three point four zero for a strengthened GRPO. Simply adding the two losses together lands between them.

The comparison that most directly tests the thesis is an ablation. Training the teacher on correctness alone lifts the self-distillation baseline from sixty-seven point zero six to sixty-nine point three five, but does worse than simply training the student with a sequence-level loss, at seventy point six one. The reason given: rewarded only for being right, the teacher learns to copy the privileged context. Add the independence reward and the mean reaches seventy-three point nine six. In the weighting ablation, weighting by raw mean reward actually hurts, and only the version that rescales and clips beats the unweighted run. Under TESD the teaching-ability measure rises over training instead of decaying, and with more rollouts TESD keeps improving, to fifty-nine point six two at forty-eight, while the earlier method stays flat.

There is a limitations section, in the appendix, and it names two things. TESD adds hyperparameters, chiefly the weight on the independence reward, and its own sweep shows the best value differs by subset. And it inherits the moving-average teacher, which smooths out and so slows the very teacher improvement the method is built to drive.

Beyond that, scores are the best accuracy reached at any evaluation during training, measured on the test split, and some splits are small: Biology has fifty questions. No error bars or repeated runs are reported. The margins on the means are around a point, and TESD does not win every column: on Olmo it trails the earlier self-distillation method on Biology by about seven points, and on the smallest Qwen model the simple combined loss edges it on coding.

The idea to carry away is that in self-distillation the teacher is a student too. The model that grades each token has the same abilities as the model being graded, plus one solution, and left alone it learns to reward resemblance to that solution. TESD gives the teacher its own rollouts and its own reward, pays it for being right without copying, and lets the student trust it only as far as those rewards suggest it should be trusted.