Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville and 1 others
KAIST AI · Microsoft · University of Toronto · Mila · Université de Montréal · CIFAR
Published September 8, 2026 · on the feed September 9, 2026 · ▲ 81 · arXiv 2609.08798
The abstract
Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important for successive model generations and multi-domain consolidation, where repeating frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduce On-Policy Reverse Distillation (OPRD), which evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches. Response-style analysis shows that OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers, suggesting that teacher guidance accelerates rather than redirects the student's own optimization. Results in conventional strong-to-weak distillation further demonstrate that OPRD effectively combines verifier-driven policy optimization with teacher guidance regardless of capacity ordering.
The Brief
2:27A short podcast summary of the paper. Written and voiced by AI.
Transcript
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation. By Youngrok Park and eight others, from KAIST AI, Microsoft, and four other institutions. Published on the eighth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
Weak-to-strong generalisation asks whether a stronger model can learn from a weaker supervisor and then surpass it. The question matters most for successive model generations, and for consolidating several domains into one model, where repeating frontier-scale post-training from scratch can cost too much. Conventional distillation makes this hard. It treats the weak teacher as the target to optimise toward, and that risks imposing the teacher's capacity ceiling on the student. A student aiming to match its teacher has no reason to go past it.
On-Policy Reverse Distillation, or OPRD, changes the teacher's role from target to direction. It looks at how the teacher's policy has shifted relative to its own reference policy, evaluated on rollouts the student produces. It then amplifies the component of the student's verifier-driven policy gradient that points along that shift.
The student is still learning from a verifier. The teacher does not set what counts as correct; it only indicates which verifier-supported updates to emphasise. Because OPRD rescales only updates the verifier already supports, the stationary points of policy optimisation are preserved: the place the student would converge to on its own does not move. What changes is the speed of getting there, and learning can continue beyond where the teacher stands.
In two settings, successive model transfer and multi-teacher distillation, OPRD reaches higher performance with fewer student updates than existing reinforcement-learning and distillation approaches. An analysis of response style shows OPRD students staying closer to models trained with verifier-based reinforcement learning alone than to their weak teachers. That suggests the teacher's guidance accelerates the student's own optimisation rather than redirecting it.
The method is not tied to the teacher being weaker. In conventional strong-to- weak distillation as well, OPRD effectively combines verifier-driven policy optimisation with teacher guidance, whatever the ordering of capacity.
Distillation usually fuses two things that OPRD pulls apart: where the student ends up, and how fast it gets there. The verifier decides the first, and the teacher only influences the second. A weak teacher can therefore speed a strong student along without dragging it down to the teacher's level.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A weak teacher, a strong student, and a ceiling nobody wants to inherit.
You have a new, larger model and an older, smaller one already post-trained on maths. Repeating frontier-scale post-training from scratch on the new model costs too much.
The road ahead
- 1 · The problem ~6 min An older, smaller model already learned maths through costly post-training, and you want a bigger new model to reuse that without being capped by it.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Use the teacher's post-training shift as a direction that boosts part of the student's own checker-driven update, never as a target to match.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each step, the student samples answers, the teacher's shift is measured on them, and the checker's gradient is stretched along that shift before updating.
- 6 · The experiment ~6 min You’ll guess: After training with OPRD, did the students' response style end up closer to their weak teachers or to models trained with verifier-based RL alone?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.