When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
Yuxiao Yang, Tianrun Yu, Shangzhe Li, Kaixiang Zhao, Xuchao Zhang, Chetan Bansal, Huaxiu Yao, Taylor W. Killian and 1 others
University of North Carolina at Chapel Hill · Brigham Young University · Microsoft
Published September 17, 2026 · on the feed September 18, 2026 · ▲ 107 · arXiv 2609.20511
The abstract
We study length inflation in on-policy distillation (OPD), where student responses can become excessively long and even exhaust the generation budget. We identify termination-token mismatch between base students and post-trained teachers as an important source of this behavior. Across Qwen3, Llama, and Gemma, the two models can place their stopping probability on different EOS tokens, even when their declared stopping sets are identical. This mismatch can suppress the student's preferred termination action without reliably transferring the teacher-preferred alternative. We show that aligning the decoding stopping set alone is insufficient, while treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigates mismatch-induced length inflation across all three model families. To further understand how termination behavior evolves over training, we study OPD across different K2-Horizon training stages. This stage-wise analysis shows that termination preferences can shift substantially during training, while also revealing a distinct length inflation late in the OPD run that persists beyond termination alignment. Together, these results identify termination mismatch as an important, but not exhaustive, source of OPD length dynamics. We release an implementation incorporating the proposed termination-handling corrections.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
When EOS Tokens Disagree. Understanding Length Inflation in On-Policy Distillation. By Yuxiao Yang and eight others, from the University of North Carolina at Chapel Hill, Brigham Young University, and one other institution. Published on the seventeenth of September. Its code repository is worked out: about nine hundred source files, with a readme.
In on-policy distillation, a student model's responses can become excessively long, sometimes long enough to exhaust the whole generation budget. This study identifies one important source of that length inflation: a mismatch in termination tokens between the base student and the post-trained teacher. The two models can place their stopping probability on different end-of-sequence tokens, even when the sets of stopping tokens they declare are identical. The mismatch can suppress the student's own preferred way of stopping, without reliably teaching it the teacher's preferred alternative. The student is discouraged from stopping its way and never quite learns to stop the teacher's.
The pattern appears across Qwen3, Llama and Gemma. Aligning the decoding stopping set alone is not enough to fix it. What does substantially mitigate the inflation, across all three model families, is treating functionally equivalent EOS tokens as one shared, semantic stopping action. That contrast is the diagnosis in miniature. The trouble is not which tokens are allowed to end a response, but which of them each model actually prefers. Fixing the allowed set leaves the preferences at odds; merging the equivalent tokens into one action removes the disagreement.
To see how termination behaviour evolves over training, the authors also study on-policy distillation across different K2-Horizon training stages. That analysis shows termination preferences shifting substantially during training. What a model prefers as a way to stop is not fixed at the start. It moves as training proceeds.
The same stage-wise analysis marks the limit of the explanation. It reveals a distinct length inflation late in the distillation run that persists even after termination is aligned. The authors therefore call termination mismatch an important, but not exhaustive, source of length dynamics in on-policy distillation. Fixing the stopping tokens removes one cause of runaway length, but not all of it.
An implementation incorporating the termination-handling corrections is released. The lesson is that two models can agree on what counts as stopping and still disagree on how to stop, and a student trained between them can pay for that disagreement in length.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a student that will not stop talking.
You are distilling a post-trained teacher into a base student, training on the student's own answers with the teacher scoring every token. Accuracy climbs.
The road ahead
- 1 · The problem ~7 min While a student model learns from a teacher on its own answers, accuracy rises but its answers keep growing until it cannot stop.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Student and teacher end answers with different end tokens; comparing their total probability of stopping, not one token, keeps stopping rewarded.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Train the student on its own answers, find where each model puts its stop probability, test decoding and signal fixes, then follow it across training stages.
- 6 · The experiment ~6 min You’ll guess: Did adding the teacher's end tokens to the student's decoding stopping set stop the runaway response length?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.