Mitigating the Length-Scaling Tax with Online Distillation
Xu Wan, Wenyue Xu, Shengjie Zhao, Mingyang Sun
ByteDance Seed · Tongji University · Peking University
Published September 30, 2026 · on the feed October 1, 2026 · ▲ 4 · arXiv 2609.38854
The abstract
Length scaling during reinforcement-learning (RL) post-training is often viewed as a sign of improved reasoning ability, especially on difficult problems, but may also make responses to already-solved problems unnecessarily verbose. We quantify this side effect as the length-scaling tax (LST): excess response length on already-solved queries without a commensurate accuracy gain. To mitigate LST, we propose Length Self-Distillation (LSD), which routes solved prompts to on-policy distillation and retains the original RL objective for unsolved prompts. LSD uses an exponential moving average of the online policy as its teacher, requiring no external model. We find that LSD achieves comparable or better performance than RL across multiple variants, while substantially curbing response-length growth on easy queries. LSD reduces LST from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks, demonstrating that LSD effectively preserves concise response patterns on easy queries while supporting efficient exploration on difficult queries during RL post-training.
The Brief
1:51A short podcast summary of the paper. Written and voiced by AI.
Transcript
Mitigating the Length-Scaling Tax with Online Distillation. By Xu Wan and three others, from ByteDance Seed, Tongji University, and one other institution. Published on the thirtieth of September. No code repository is linked to it.
When responses grow longer during reinforcement-learning post-training, that growth is often read as a sign of better reasoning, particularly on hard problems. The authors point to a side effect. Answers to problems the model already solves can become needlessly verbose as well. They put a name and a number on it: the length-scaling tax, defined as excess response length on already-solved queries that comes without a matching gain in accuracy.
Length Self-Distillation, or LSD, treats solved and unsolved prompts differently. Prompts the model already solves are routed to on-policy distillation. Prompts it does not yet solve keep the original reinforcement-learning objective. Easy questions are pulled toward concise behavior, while hard ones stay free to explore.
The teacher for that distillation is not a separate model. LSD uses an exponential moving average of the online policy itself, so no external model is required.
Across multiple variants, LSD performs comparably to plain reinforcement learning or better, while substantially curbing how much responses to easy queries grow. On single-turn reasoning, it brings the length-scaling tax from nineteen percent down to minus three point seven percent. On multi-turn agentic tasks, it brings the tax from thirty-one point four percent down to thirteen point seven percent.
On single-turn reasoning the tax does not just shrink. It drops below zero, so by the paper's own measure the excess length on solved queries is gone in that setting. Multi-turn agentic work keeps some of its tax, though less than half of what it was. The design that gets there is simple to state: distil where the model already succeeds, and keep exploring where it does not.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Longer answers look like better reasoning. On easy questions, they may just be longer.
You are post-training a reasoning model with reinforcement learning, and its responses keep getting longer. On hard problems, that may be real extra thinking.
The road ahead
- 1 · The problem ~7 min Reinforcement learning makes answers longer everywhere, including on problems already solved, and the usual reward gives no signal to keep those easy answers short.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Send prompts the model already solves to distillation from a lagging copy of itself, and keep ordinary RL for the prompts it still fails.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each step: sample answers with the current policy, route each prompt by whether its group is solved, then distill solved prompts and apply RL to the rest.
- 6 · The experiment ~6 min You’ll guess: On multi-turn agentic tasks, plain RL showed a length-scaling tax of 31.4%. What tax remained with Length Self-Distillation?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.