Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
Run Yang, Runpeng Dai, Jie Sun, Jielei Zhang, Fan Zhou, Hongtu Zhu, Peiyi Li, Longwen Gao
BiliBili.Inc · University of North Carolina at Chapel Hill · University of Science and Technology of China · Shanghai University of Finance and Economics
Published August 30, 2026 · on the feed September 3, 2026 · ▲ 15 · arXiv 2608.29846
The abstract
Sampled-token on-policy distillation (OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teacher probabilities only for sampled tokens. Yet it frequently suffers from diversity distillation failure: the student's pass@1 improves while its pass@k plateaus, failing to inherit the teacher's diversity. To explain this, we introduce First-Order Local Entropy Influence, a signed first-order proxy that decouples each update's entropy effect into the teacher--student log-probability gap and the student's local probability structure, and empirically links entropy contraction to negative-influence positions. Motivated by this, we propose Influence-Directed Adaptive On-Policy Distillation (IDA-OPD): rather than relying on costly full-vocabulary Forward-KL objectives, it preserves entropy-expanding updates while replacing entropy-contracting ones with divergence-adaptive advantage shrinkage, using only the teacher's sampled-token log-probability. Experiments on reasoning-oriented distillation show IDA-OPD consistently improves pass@k, inheriting the teacher's diversity through distillation, matches the strongest teacher-informed methods at strictly lower cost, and broadly maintains vanilla OPD's pass@1, all without full-vocabulary teacher information.
The Brief
2:18A short podcast summary of the paper. Written and voiced by AI.
Transcript
Influence-Directed Distillation. Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation. By Run Yang and seven others, from Bilibili, the University of North Carolina at Chapel Hill, and two other institutions. Published on the thirtieth of August. No code repository is linked to it.
Sampled-token on-policy distillation is an efficient way to pass capabilities from a teacher model to a student. The student generates the tokens, and the teacher only has to supply probabilities for the tokens actually sampled. It has a recurring failure, which the authors call diversity distillation failure. The student's pass at one improves, while its pass at k stalls. The student gets better at its single best answer, but does not inherit the teacher's diversity across many attempts.
To explain why, the paper introduces First-Order Local Entropy Influence. It is a signed, first-order proxy for how each update changes the student's entropy, and it splits that effect into two parts: the gap between teacher and student log-probabilities, and the student's local probability structure. Applied empirically, it ties entropy contraction to positions where the influence is negative. Some updates widen the student's distribution; others narrow it, and the narrowing ones are where diversity is lost.
That diagnosis motivates Influence-Directed Adaptive On-Policy Distillation, or IDA-OPD. It does not rely on full-vocabulary Forward-KL objectives, which are costly. IDA-OPD instead keeps the updates that expand entropy as they are, and replaces the ones that contract it with divergence-adaptive advantage shrinkage. It does this using only the teacher's log-probability for the sampled token, so the cheap regime of the original method is kept.
Experiments are on reasoning-oriented distillation. IDA-OPD consistently improves pass at k, so the student does inherit the teacher's diversity. It matches the strongest teacher-informed methods at strictly lower cost. And it broadly maintains the pass at one of vanilla on-policy distillation. All of that holds without any full-vocabulary information from the teacher.
Two quantities were moving in different directions, and the contribution is to find where in training they split. With a signed measure of each update's effect on entropy, the method can tell which updates to trust and which to temper, rather than paying for the whole vocabulary to fix everything at once.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The student gets sharper and narrower at once. Something in training is draining its variety.
You are training a small student model from a big teacher. The student writes its own answers, and the teacher only says how likely it finds each token the student picked. Cheap and effective.
The road ahead
- 1 · The problem ~6 min A cheap way of teaching a small model from a big one improves its best answer but fails to pass on the teacher's variety.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Flag each update by whether it narrows or widens the student's distribution, then shrink only the narrowing ones, using one teacher number per token.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Sample from the student, score each token with the teacher, compute each update's entropy influence, then shrink only the contracting updates.
- 6 · The experiment ~6 min You’ll guess: Protecting diversity can cost single-answer accuracy. What happened to the student's pass@1 under IDA-OPD compared with vanilla OPD?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.