From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

From Gradients to Capabilities. Understanding Multi-Teacher On-Policy Distillation. By Siqi Zhu and six others, from the University of Illinois Urbana-Champaign, Princeton University, and one other institution. Published on the first of October. No code repository is linked to it.

Multi-teacher on-policy distillation, or MOPD, aims to combine the strengths of several teachers trained with reinforcement learning inside a single student. How the teachers' signals actually turn into parameter changes has been underexplored. The claim here is that several factors shape those signals along the way: how the loss is averaged, the optimizer's first moment, the precision the weights are stored in, and the choice of gradient.

The study uses Qwen3-1.7B with four domain teachers, each trained with RL from the same initialization as the student. It compares gradients, optimizer updates and task learning curves, with additional diagnostics on SmolLM3-3B. First, loss averaging implicitly weights responses. Token averaging favors longer responses, and equalizing the contributions of domains keeps that weighting inside each domain.

Second, Adam's first moment reduces the differences in parameter updates. Even though raw gradients differ, the cosine similarity of updates is zero point eight three between teachers, and zero point nine six between averaging rules. Third, BF16 rounding hides small changes. About ninety-seven percent of FP32 master weights differ from initialization, yet only seven to eleven percent of BF16 weights do.

Finally, the top sixty-four intersection KL gradient closely matches Qwen's full-vocabulary gradient, but its effect on task performance depends on averaging. Under response averaging, mathematics accuracy is two point six points higher than with sampled-token policy-gradient. Under global token averaging, it is two point one points lower.

The setting is specific. The main subject is one student with four domain teachers sharing its initialization, plus diagnostics on a second model. The last finding is itself a lesson about scope: the same gradient helps or hurts depending on an averaging choice, so none of these effects should be read apart from the configuration around it.

What a teacher signals is not quite what the student's weights receive. Averaging rules, optimizer state and number formats all sit between the two, and each can reweight, blur or erase part of the signal before it ever becomes a capability.