PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation
Youzhi Liu, Ruobing Zheng, Boyuan Tong, Tianqi Li, Pingqi Li, Hanbo Bi, Yi Yuan, Jingdong Chen
Ant Group
Published September 28, 2026 · on the feed September 30, 2026 · ▲ 5 · arXiv 2609.34605
The abstract
Multi-teacher on-policy distillation (MOPD) has emerged as a popular post-training paradigm for integrating specialized capabilities in frontier language models. Existing OPD research has primarily focused on optimizing single-task distillation through objective design, distillation scope, and teacher signal construction, whereas MOPD must aggregate multiple capabilities in shared parameters and address the resulting capability seesaw, in which improving one domain suppresses capabilities acquired from another. Inspired by the distinctive update geometry of OPD, we find that parameter updates from different tasks rapidly concentrate in their respective low-dimensional subspaces during MOPD, providing a direct geometric basis for identifying and controlling cross-task interference. We therefore propose PMOPD (Projection-based Multi-Teacher On-Policy Distillation), which constructs subspace memories from the cumulative parameter displacements of different tasks and projects both gradients and optimizer updates to remove components that interfere with protected task directions. We further develop a lightweight conflict probe to characterize task interactions and guide task ordering, together with a cycling strategy that balances subspace estimation and timely task revisitation. Experiments on representative Code, Reason, and Math tasks show that PMOPD improves every evaluated capability over MOPD, raising the average score across the three tasks by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B. These consistent gains establish geometry-aware optimization as an effective and transferable approach to balanced multi-teacher distillation.
The Brief
2:44A short podcast summary of the paper. Written and voiced by AI.
Transcript
PMOPD. Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation. By Youzhi Liu and seven others, from Ant Group. Published on the twenty-eighth of September. No code repository is linked to it.
Multi-teacher on-policy distillation, MOPD for short, has become a popular way to post-train frontier language models: several specialist teachers, one student, and the goal of folding all their capabilities into it. Most research on on-policy distillation has looked at the single-task case, tuning the objective, the scope of distillation and how the teacher signal is built. The multi-teacher case has a problem of its own. Every capability has to live in the same shared parameters, and that produces a capability seesaw. Improving one domain suppresses what the student had learned from another. Push one skill up and another sinks; restore that one and the first slips back down. Better single-task objectives do not address this, because the trouble lies in how the tasks share space, not in how any one of them is taught.
The way in is geometric. On-policy distillation has a distinctive update geometry, and the authors find that during MOPD the parameter updates from each task rapidly concentrate in their own low-dimensional subspace. If each task leaves its mark in a small region of parameter space, then interference between tasks becomes something that can be located, and once located, it can be controlled.
PMOPD, short for Projection-based Multi-Teacher On-Policy Distillation, acts on that directly. It builds subspace memories from the cumulative parameter displacements of each task. Then it projects both the gradients and the optimizer updates so that components interfering with the protected task directions are removed. Two supporting pieces decide the schedule. A lightweight conflict probe characterizes how tasks interact and guides the order in which they are trained. A cycling strategy balances two needs that pull against each other: estimating the subspaces well, and coming back to each task in time.
The experiments use representative Code, Reason and Math tasks. PMOPD improves every evaluated capability over plain MOPD. The average score across the three tasks rises by two point five four points on Qwen2.5-7B, and by two point zero nine points on Llama-3.1-8B. The gain shows up on both model families, not only one. And the improvement is not a trade, which is the point for a method aimed at a seesaw: no evaluated capability was given up to pay for the others.
Picture each teacher's skill as a narrow channel cut through the same block of parameters. The seesaw happens when one task's updates spill into another's channel. Measure where the channels are, keep later updates out of them, and pick the order of work with a probe, and the authors argue you have a geometry-aware recipe for balanced distillation that transfers across models.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Three teachers, one student, and a seesaw: every gain in one skill seems to cost another.
You are folding three specialist teachers, for maths, reasoning and code, into one student model. Each teacher scores tokens on answers the student samples itself.
The road ahead
- 1 · The problem ~6 min When several specialist teachers train one shared student, gains in one skill keep wiping out gains in another.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Find the few directions each task's training actually moved the weights, then stop later tasks from moving along them.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Probe conflicts to set the order, then for each task block record the weight change, store its directions, and clean later updates.
- 6 · The experiment ~6 min You’ll guess: Compared with plain MOPD, by how many points did PMOPD raise the average score across Code, Reason and Math on Qwen2.5-7B?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.