Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts

Composing What Each Teacher Learned. Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts. By Hejian Sang and five others, from Iowa State University, LinkedIn, and one other institution. Published on the seventh of October.

Multi-teacher on-policy distillation, abbreviated MOPD, shows up in two settings. In common-domain composition, several teachers score each rollout a student produces from a single prompt domain, and their signals are combined into one target. In routed-domain distillation, prompts from different domains are each sent to the matching specialist teacher.

Both settings usually transfer the teacher's endpoint policy: the teacher as it finally is. The authors point out what that mixes together. Part of a teacher's behaviour is what post-training changed. Part of it is preferences inherited from the base model the teacher started from. Endpoint transfer hands the student both, undifferentiated.

Delta MOPD transfers only the change. For each teacher it takes the teacher-minus-base logit shift, the difference between the teacher and its own base, and re-anchors that shift at the student's frozen initialization. What the student receives is what each teacher learned, placed on top of where the student itself began.

The comparison is set up to isolate that one choice. Delta MOPD is compared with endpoint supervision in both settings while teacher selection is held fixed, so only the construction of the target differs.

First the authors expose the mechanism that impedes endpoint transfer. Inherited base pull can exceed the post-training shift: the part of the teacher that came from its base can outweigh the part post-training added. Removing it reduces both the teacher-term norm ratio and the KL divergence between target and student.

The results suggest shift targets help most when teacher signals are combined at a state. Composing three teachers, delta MOPD exceeds endpoint composition by four point one one points on Math, and by one point nine five points across five benchmarks. With two teachers, it matches endpoint accuracy.

Under phased routing it achieves higher mean performance in both phase orders, and it shrinks the observed gap between orders from ten point five zero points to six point four two. Under interleaved routing, where each update involves a single teacher, the two targets perform comparably. The authors read the phased results as supporting evidence that the benefit may extend to signals accumulated across training phases.

Their conclusion is that target construction is a design axis of its own in multi-teacher distillation, independent of and complementary to the question of which teachers to choose. The pattern across settings is consistent with that reading: where signals from several teachers meet, separating learned shift from inherited base pays off, and where only one teacher acts at a time, the difference fades.