P^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors
Sinuo Wang, Jiacheng Lu, Zeyu Zhang, Maojia Song, Rui Sun, Beidi Luan, Yutong Xie, Jing Li and 3 others
Adelaide University · StepFun · Shanghai Jiao Tong University · Singapore University of Technology and Design · Mohamed bin Zayed University of Artificial Intelligence
Published November 2, 2026 · on the feed November 9, 2026 · ▲ 64 · arXiv 2611.03953
The abstract
Multi-teacher on-policy distillation consolidates expert capabilities by supervising student-generated trajectories, whereas task arithmetic offers a training-free alternative through linear parameter combinations. We view the linear combination of student and expert parameters as a single differentiable model and propose P^2-MOPD, which jointly optimizes the mixing coefficients and the student weights under the same on-policy distillation objective. Unlike task arithmetic, which stays within the span of the frozen student and expert parameters for any choice of merging coefficients, P^2-MOPD keeps the student trainable, so the experts act as parameter priors without confining it to this span. Across four domains at 1.7B and 4B scales, P^2-MOPD achieves faster and more balanced recovery than MOPD from the same initialization, surpassing its final average recovery in half the training steps. At the end of training, our 1.7B model reaches 94.7% average recovery, versus 80.9% for MOPD.
The Brief
1:42A short podcast summary of the paper. Written and voiced by AI.
Transcript
P^2-MOPD. Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors. By Sinuo Wang and ten others, from Adelaide University, StepFun, and three other institutions. Published on the second of November.
There are two familiar ways to fold several experts into one model. Multi-teacher on-policy distillation, or MOPD, consolidates their capabilities by supervising trajectories that the student generates itself. Task arithmetic is the training-free alternative: it combines parameters linearly. The second has a limit. Task arithmetic, whatever merging coefficients it picks, stays within the span of the frozen student and expert parameters.
The proposal, P squared MOPD, puts the two together. It views the linear combination of student and expert parameters as one differentiable model, and optimizes the mixing coefficients and the student weights jointly, under the same on-policy distillation objective.
The difference from task arithmetic is that the student stays trainable. The experts still enter through the combination, but they act as parameter priors rather than as walls. The student starts with their help and is not confined to the span they define.
The experiments cover four domains, at two scales: one point seven billion parameters, and four billion. Starting from the same initialization as MOPD, P squared MOPD recovers faster and in a more balanced way, and it surpasses MOPD's final average recovery in half the training steps. At the end of training, the smaller model reaches ninety-four point seven percent average recovery, against eighty point nine percent for MOPD.
The two older approaches each kept something the other lacked. Distillation kept the student learning, and arithmetic made use of the experts' weights directly. Treating the merge itself as a trainable model lets one method keep both.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Many experts, one student, and two familiar recipes that each fall short.
You have several domain experts, each post-trained from the same base model, and you want one generalist that keeps all their skills.
The road ahead
- 1 · The problem ~6 min Combining several domain experts into one generalist is slow and uneven by distillation, and cheap but stuck by weight merging.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Treat student weights plus a learnable mix of frozen expert weights as one model, and train both parts under the distillation objective.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Write each projection as student weights plus coefficient-scaled expert task vectors, distil on the student's own rollouts, and update both channels together.
- 6 · The experiment ~6 min You’ll guess: Starting from the same initialization, how quickly did P^2-MOPD pass the final average recovery that MOPD reached at the end of training?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.