Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian
Brigham Young University · University of North Carolina at Chapel Hill
Published October 14, 2026 · on the feed October 15, 2026 · ▲ 80 · arXiv 2610.20572
The abstract
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
The Brief
1:45A short podcast summary of the paper. Written and voiced by AI.
Transcript
Recovering Reliable Supervision via Critique-Guided On-Policy Distillation. By Kaixiang Zhao and six others, from Brigham Young University and the University of North Carolina at Chapel Hill. Published on the fourteenth of October. Its code repository is public and real: about thirty source files.
On-policy distillation, or OPD, trains a student model on its own rollouts, with a teacher supplying supervision at the level of individual tokens. Built into it is an assumption: from any prefix the student writes, the teacher can still reach a correct answer. The authors call this property the recoverability of the teacher. They show it fails on incorrect rollouts. Once the student has made an error, the teacher follows it, even though the teacher recognises the error. On incorrect rollouts, then, the supervision that OPD relies on stops being reliable.
CriticOPD uses that gap between recognising an error and acting on it. If the teacher can recognise a mistake, it can be asked to point it out, and that restores recoverability.
For an incorrect rollout, the teacher writes a critique. The student then rewrites its solution following the error, guided by that critique. The teacher scores the resulting output with the critique included, as privileged information. The student, however, trains on the revision without the critique. So the teacher sees the full context when judging, while the student learns from a version that does not depend on being told where it went wrong.
The method was tested with two Qwen3 students on six mathematical reasoning benchmarks. CriticOPD improves on OPD, and the largest gains come on competition problems.
The diagnosis carries most of the weight. A teacher that can spot an error but still continues from it is a teacher whose knowledge is not reaching the student. Turning that recognition into a written critique, and keeping the critique out of what the student trains on, is the whole intervention.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A teacher that follows its student off a cliff is not much of a teacher. Start there.
You are training a small student model on its own attempts at maths problems, with a stronger teacher scoring every token it writes.
The road ahead
- 1 · The problem ~7 min When a student model trains on its own maths attempts, the teacher's token scores become unreliable once the student has made a mistake.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Let the teacher point out the student's error, have the student rewrite from there, and score the rewrite with the teacher seeing the critique.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Sample student attempts, critique the wrong ones, have the student rewrite from the last error, then score with the critique and train without it.
- 6 · The experiment ~6 min You’ll guess: Across the maths benchmarks, where did CriticOPD improve most over standard on-policy distillation?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.