Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation

Every Token Needs Its Own Teacher. Multi-teacher Adaptive On-policy Self-distillation. By Yifu Luo and five others, from the Technical University of Munich, the Munich Center for Machine Learning, and five other institutions. Published on the twenty-ninth of October.

On-policy self-distillation, or OPSD, has proven effective for post-training large language models. It constructs a teacher by conditioning the model on privileged references. The limitation lies in how fixed that teacher is. Some approaches keep it fixed throughout training, consistently exposing it to the full reference. Others vary how much of the reference the teacher sees, but only from one response to the next, so within any single response the teacher never changes. The authors argue that neither is sufficient.

Their position is the title: every token needs its own teacher. Which teacher is preferred can vary token by token within the same response, so supervision should adapt at exactly that granularity. Each earlier choice fixes the teacher over some span: the whole of training in one case, a whole response in the other. The proposal is to fix it over no span at all, and let the choice be made afresh for every token.

They ground this first with a pilot study that intervenes on the teacher. The study does two jobs. It validates the premise, that teacher preference really does differ across tokens. And it shows the premise can be acted on: a preferred teacher can be identified for each token, through majority voting and through KL-based selection. Building on those findings, they propose Multi-teacher Adaptive On-policy Self-Distillation, or MTA-OPSD, described as the first token-level adaptive OPSD method. It constructs a pool of candidate teachers, and during training every token adaptively selects its preferred teacher from that pool.

The experiments use Qwen3 models on mathematical reasoning tasks. MTA-OPSD consistently outperforms existing OPSD approaches, with an average gain of up to five point nine over the base model. The authors read this as evidence for the importance of adapting teacher supervision at the token level, rather than over a whole run or a whole response.

Earlier methods asked how much of the reference a teacher should see, and answered once per run or once per response. This one asks which teacher should supervise each token, and lets the answer change partway through a sentence. The results point to the granularity of teacher supervision as something that matters, all the way down to the single token.

Every Token Needs Its Own Teacher. Multi-teacher Adaptive On-policy Self-distillation. By Yifu Luo and five others, from the Technical University of Munich, the Munich Center for Machine Learning, and five other institutions. Published on the twenty-ninth of October. This is the long version: the whole paper rather than the abstract.

On-policy self-distillation, called OPSD, trains a language model to be its own teacher. The student writes a response. The same model, now shown a privileged reference such as a worked solution, scores every token of that response, and the student is pulled toward the teacher's next-token distribution. Compared with reinforcement learning from verifiable rewards, the signal is dense, one per token instead of one per answer. Compared with supervised fine-tuning, the student learns on its own samples. The trouble, as several recent studies report, is that a teacher holding the whole solution can be far too sure of itself. Its distribution turns overly sharp, it suppresses uncertainty, and the supervision stops matching what the student is able to learn. Reasoning can actually get worse.

Earlier fixes adjusted how much of the reference the teacher sees, based on something like the gap between student and teacher. But those methods adapt once per response. Every token in a rollout still answers to one teacher. The authors' claim is that this is the wrong granularity: every token needs its own teacher.

To test that, they ran a teacher-intervention pilot study. Take a student response and pick a position inside it. Keep the student's prefix, let a candidate teacher write a short continuation from that point, then hand control back to the student to finish. Compare the accuracy of this student, teacher, student relay with the student alone. The gain is a proxy for how well that teacher suits that position. They tried seven positions per rollout, some evenly spaced and some chosen for extreme entropy or KL distance, and seven teachers: the full reference, and prefix, suffix or random-span truncations of it at exposure ratios of one half and three quarters. In all, seventy-five thousand two hundred and sixty-four generations were evaluated, on Qwen3-1.7B and Qwen3-4B.

The pilot also suggested how to pick a teacher. At each token, every teacher casts a vote: does it assign the student's sampled token more probability than the student did, or less? That log-probability ratio is the advantage signal, and its sign is the vote. In eighty to ninety percent of cases the best teacher sat in the majority, and majority teachers gained more on average than minority ones. Within the majority, the teacher farthest from the student in KL distance beat the nearest one at every regular position tested.

MTA-OPSD turns this into a training rule. It builds a pool of seven teachers, all the same model, differing only in which slice of the reference they are conditioned on. For each token separately, it takes the majority vote over the pool, discards the dissenters, and among the remaining teachers selects the one with the largest KL distance from the student. The loss is then ordinary self-distillation, except the teacher in each token's KL term is that token's own pick.

The ordering is the point, according to the authors. Voting settles direction: should this token be reinforced or pushed down? KL then picks the most informative teacher that agrees with that direction. When the majority supports the token, the farthest teacher still reinforces it while carrying a different spread over the alternatives, so the student is not simply copying itself. When the majority opposes it, the farthest teacher moves more probability onto other candidates, giving a stronger correction than a teacher that barely disagrees. KL distance alone is never treated as a measure of quality.

Training uses DeepMath-103K, with evaluation on AMC23, AIME24 and AIME25, at three Qwen3 sizes. MTA-OPSD has the best average at each size: fifty-seven point zero for Qwen3-1.7B against fifty-four point zero for plain OPSD and for Paint, the response-level adaptive baseline. The gains over the base model are five point nine, one point five and two point eight points, and they are clearest on AIME25.

The ablations are the sharper test. Training on any single teacher from the pool tops out at fifty-six point two, with prefix truncation at three quarters. Keeping the whole pool but picking a teacher at random per token gives fifty-three point six, and averaging the loss over all teachers gives fifty-three point seven. Both are below that best single teacher. So a pool alone does not help; the selection rule does. Removing majority voting drops the score to fifty-two point nine, and choosing the nearest rather than the farthest teacher gives fifty-three point three. Choosing at random within the majority costs much less, landing at fifty-six point six.

There is no limitations section. The conclusion names two directions for future work, self-evolving teacher pools and more stable multi-teacher training, the second of which hints at instability without reporting any. Beyond that, the evidence has visible edges. Each method is trained for three hundred steps, checkpoints are evaluated every fifty steps on the test set, and the best one is reported, with no error bars or repeated seeds. At Qwen3-4B, MTA-OPSD improves the average by under a point over plain OPSD, and on AMC23 at the two larger sizes it does not take the top score. All ablations use only the smallest model, and the evaluation covers mathematical reasoning alone.

One quieter observation reframes the whole line of work. A higher exposure ratio did not reliably produce a teacher farther from the student: under suffix truncation, the three-quarter teacher was the more distant one in only forty-nine point one percent of cases. So simply dialling up or down how much of the solution the teacher sees is a weak handle on supervision. The idea to carry away is that the right teacher is a property of the token, not the response, and can be found cheaply at training time: let the teachers vote on the direction, then take the boldest one that agrees.