Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD
Yibo Zhao, Zixuan Yang, Yunshi Lan, Xiang Li
East China Normal University
Published September 26, 2026 · on the feed October 1, 2026 · ▲ 5 · arXiv 2609.29142
The abstract
Direct On-Policy Distillation (Direct-OPD) transfers reinforcement-learning-induced policy improvements from a small model to a larger student by using the token-level log-ratio between post-RL and pre-RL checkpoints as dense supervision on the student's own rollouts. This transfer rewards the policy shift at every state, yet the log-ratio measures only relative change: it can stay fixed even as the probability mass that both checkpoints assign to the student's candidate tokens vanishes. Through an exact construction, we show that the Direct-OPD reward and its update can remain unchanged while the Jensen-Shannon divergence (JSD) and both KL directions between the checkpoints vanish with this mass, and we note that a small JSD bounds how much the teacher's behavior changed. Motivated by this analysis, we propose Selective Supervision for Direct-OPD (S^2D-OPD), which ranks student-sampled states by their teacher-reference JSD and masks Direct-OPD supervision at low-divergence states, retaining only the top 10% of states per response. Across two teacher pairs and four student models ranging from 1.7B to 8B parameters, S^2D-OPD improves held-out accuracy over dense Direct-OPD on AIME and HMMT benchmarks in seven of eight settings and matches it in the eighth, without extra forward passes.
The Brief
2:26A short podcast summary of the paper. Written and voiced by AI.
Transcript
Not Every Token Is Worth Distilling. Selective Supervision for Direct-OPD. By Yibo Zhao and three others, from East China Normal University. Published on the twenty-sixth of September. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.
Direct On-Policy Distillation, Direct-OPD, moves the improvement that reinforcement learning produced in a small model over to a larger student. It does this with a token-level log-ratio between post-RL and pre-RL checkpoints, used as dense supervision on the student's own rollouts. Every state gets rewarded for the policy shift. But a log-ratio only measures relative change. It can hold steady even as the probability mass that both checkpoints place on the student's candidate tokens drains away to nothing.
The authors make that concrete with an exact construction. The Direct-OPD reward and its update stay unchanged, while the Jensen-Shannon divergence, JSD, and both directions of KL between the checkpoints vanish along with that mass. A small JSD, they note, bounds how much the teacher's behavior actually changed. In those states the reward looks just as strong as anywhere else, while the teacher it is supposed to reflect has barely changed at all. The student is being pushed with full force by a difference that no longer describes much.
Selective Supervision for Direct-OPD, written S two D OPD, follows directly from that analysis. Rather than supervising everywhere, it supervises only where the teacher's behavior genuinely moved. The log-ratio still supplies the signal. Divergence decides whether the signal is used at all.
For each response, the states the student sampled are ranked by their teacher-reference JSD. Direct-OPD supervision is masked at the low-divergence states, and only the top ten percent of states per response are kept. All of this comes without extra forward passes.
The test spans two teacher pairs and four student models, from one point seven billion up to eight billion parameters. On AIME and HMMT benchmarks, S two D OPD improves held-out accuracy over dense Direct-OPD in seven of eight settings, and matches it in the eighth. In none of the eight does throwing away most of the supervision leave the student worse off than the dense baseline.
A dense signal is only as good as the states it lands on. When the log-ratio says the policy shifted but the divergence says almost nothing changed, the supervision there carries little about the teacher. Keeping one state in ten was enough to improve on the dense version in all but one setting, and to tie it in that one.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A free lunch from a small model sounds good. The question is whether every bite is worth taking.
You want a large model to gain what reinforcement learning gave a small one, without paying for RL at the large scale. A neat trick exists: compare the small model before and after RL, token by token, and reward the large student for that shift.
The road ahead
- 1 · The problem ~7 min Copying a small model's RL improvement into a large student with a log-ratio reward can push hard at states where the teacher barely changed.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Keep the log-ratio reward only at the states where the teacher's whole distribution moved most, measured by teacher-reference divergence, and mask the rest.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Sample student answers, score each state by teacher-reference JSD, rank within each response, mask the low ones, and apply the log-ratio reward to the rest.
- 6 · The experiment ~6 min You’ll guess: Keeping supervision on only a small share of states per response, how did the method fare against dense Direct-OPD across eight teacher-student settings?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.