Calibrating Teacher--Student Discrepancy for On-Policy Distillation
Qiangqiang He, Jin Li, MingCai Chen
Nanjing University · Southeast University · Nanjing University of Posts and Telecommunications
Published September 18, 2026 · on the feed September 21, 2026 · ▲ 13 · arXiv 2609.21619
The abstract
On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce Calibrated On-Policy Distillation (Cal-OPD), which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65\% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.
The Brief
1:57A short podcast summary of the paper. Written and voiced by AI.
Transcript
Calibrating Teacher--Student Discrepancy for On-Policy Distillation. By Qiangqiang He and two others, from Nanjing University, Southeast University, and one other institution. Published on the eighteenth of September. No code repository is linked to it.
On-policy distillation improves reasoning models by having an on-policy student learn the token-level discrepancy between itself and a stronger teacher. That discrepancy is not purely a capability gap. It also contains deviations arising from the teacher itself, and those get mixed into the observed difference and learned indiscriminately by standard distillation. Privileged on-policy distillation makes this worse. Privileged information induces larger likelihood shifts on the teacher's side, so the student is pushed to learn even more of the teacher's own deviation.
Calibrated On-Policy Distillation, Cal-OPD, separates the two parts. It estimates a region that represents the teacher's self-deviation, then calibrates the original teacher-student discrepancy by keeping only the component that lies beyond that region.
The estimate comes from privileged interventions made in two directions, one positive and one negative. Together they mark out how far the teacher moves on its own. Whatever part of the discrepancy falls inside that region is treated as the teacher's own deviation, and set aside. Only what lies outside it is kept as the signal the student learns from. The student still learns from the teacher. It simply stops learning the portion of the difference that the teacher would have produced on its own.
On mathematical reasoning benchmarks, Cal-OPD keeps only about fifty-two to sixty-five percent of the original discrepancy as its optimization signal, and still consistently outperforms standard on-policy distillation and its variants across model scales.
Less signal turns out to be better signal here. By discarding the part of the teacher's difference that reflects the teacher's own movement rather than its advantage, Cal-OPD trains on a smaller difference that sits closer to the capability gap distillation was meant to transfer.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A teacher's scores look like solid ground. Here they wobble.
You are training a small reasoning model on its own answers, with a stronger teacher scoring every token it writes. The student learns from the gap between the teacher's likelihood and its own.
The road ahead
- 1 · The problem ~7 min A student model learns from the gap between its scores and a teacher's, but part of that gap is the teacher's own wobble.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Measure how far the teacher's scores drift on their own, and let the student learn only the part of the gap beyond that drift.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Student writes answers, the teacher re-scores them under contrasting hints, and the student learns only the gap outside the teacher's measured range.
- 6 · The experiment ~6 min You’ll guess: Cal-OPD learns from only part of the original teacher-student discrepancy. How did it fare against standard on-policy distillation on maths reasoning?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.