Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

Latent-MOPD. Latent Multi-Teacher On-Policy Distillation. By Zhengyu Fang and six others, from Case Western Reserve University, Zillow Group, and two other institutions. Published on the first of October. Its code repository is a placeholder: a readme, and nothing to run yet.

On-policy distillation, or OPD, trains a student on the responses the student itself generates. When there are several teachers, each a specialist, existing multi-teacher OPD for language models transfers one kind of thing from them: what they predict, carried through their output distributions. Everything the specialists computed on the way to those predictions is left behind. The hidden states that produced each answer never reach the student.

Latent-MOPD takes both channels. To the authors' knowledge it is the first representation-level multi-teacher OPD method for large language models. It integrates existing specialists through their predictions and through the hidden states used to compute them, and it does so without any additional training of the teachers.

Supervising one student with the internal representations of several specialists raises coordination problems, and the method answers each one in turn. Targets are taken from late layers, chosen according to the relationship between teacher and student. Teachers whose hidden widths differ from the student's are bridged by a shared projection. Updates are grouped by domain. Across training, each teacher's supervision shifts gradually from hidden states toward token predictions, and both channels draw on the same routed specialist, so a student is never being pulled by one teacher's representations and another teacher's outputs at once.

In the main setting, where teachers and student come from the same family, Latent-MOPD beats token-only, representation-only and uniform-averaging baselines on all nine benchmarks, spanning math, code and logic. The student has the same parameter count as each teacher, and it surpasses the best teacher for a given benchmark on a majority of them. With larger cross-family teachers that were developed separately, it still beats both single-channel baselines on every benchmark.

One control shows why the grouping matters. A same-family, all-layer, representation-only variant stays stable when each update is domain-pure, and collapses when teacher domains are interleaved within an update.

The claim the authors draw is that a single student can absorb capabilities from several specialists through two routes at once: the distributions they output and the representations they hold inside. The collapse under mixed domains is the cautionary half of the same result. Reaching into a teacher's hidden states is useful, but only when the student is not asked to follow several of them inside one step.