Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Beyond Teacher Assignment. Domain-Normalized Multi-Teacher On-Policy Distillation. By Xin Li and eight others, from Nanyang Technological University, Yale University, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about four hundred source files, with a readme and a licence.

Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions. Users, though, need one model with all of those skills. Multi-teacher on-policy distillation, or MOPD, merges them by letting the specialists teach one student. The student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. That routing decides which specialist teaches. It does not decide how strongly its feedback moves the shared student.

The gap shows up in practice. In Qwen3.5 models at three sizes, the MOPD student does not beat a student taught by the best single specialist, and it gains little of the mathematics specialist's advantage. The authors trace this to unbalanced feedback. Instruction-following feedback is several times more spread out than mathematics feedback, and it dominates the student's updates.

Domain-Normalized MOPD, or DN-MOPD, keeps the routing exactly as it is and adds one step. Each domain's feedback is rescaled by its measured spread.

Routing still decides who teaches. Normalization decides how loudly each teacher is heard, so a domain whose feedback happens to be more spread out no longer drowns out the others in the shared update. Nothing else about the distillation recipe needs to change. The diagnosis explains why this is the natural fix. If one domain's feedback is several times more spread out than another's, then on a shared set of weights it will push harder simply by being larger, whatever its actual value to the student. Rescaling by spread puts the domains on a common footing before their feedback is combined.

On six public benchmarks, DN-MOPD improves the average score over MOPD at every model size, across three random seeds, and under two limits on answer length. It also recovers most of the mathematics gain that plain MOPD had lost. The robustness checks matter here, because a gain that held for one seed or one length limit could be noise, and this one holds across each of them.

Controls with fixed domain weights show where the improvement comes from. The gain comes mainly from turning instruction-following feedback down, rather than from turning mathematics feedback up on its own. And fixed weights close to the ones DN-MOPD measures perform comparably. Read together, those controls say the benefit lies in reaching the right balance between domains, and that the spread DN-MOPD measures lands close to that balance on its own.

Merging specialists, on this account, involves two decisions rather than one. Routing had settled which teacher speaks. The second decision, how much each teacher's feedback counts, had been left to the spread of the feedback itself, and in that contest instruction following was winning by default.