Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction
Junxian Li, Ruixuan Yang, Tianao Zhang, Tiange Xu, Weisheng Dong, Yulun Zhang
Shanghai Jiao Tong University · Xi'an Jiaotong University · University of Cambridge · Xidian University
Published September 26, 2026 · on the feed September 29, 2026 · ▲ 8 · arXiv 2609.32353
The abstract
Visual token reduction is an effective way to accelerate multimodal large language models (MLLMs), but performance deteriorates rapidly under extremely low token budgets. Existing work has explored both visual-token selection and training-based adaptation to reduced visual inputs. We take a step further by asking how a heavily compressed MLLM should learn from the states induced by its own generations. This setting naturally calls for on-policy self-distillation: a heavily compressed model is supervised on the states induced by its own generations, while its full-token counterpart serves as an information-rich teacher. Based on this insight, we propose LT-OPD, a training framework for extreme visual-token reduction. The student rolls out responses with only a small fraction of visual tokens, and a frozen full-token copy of the same MLLM provides distributional supervision along these student-generated trajectories. To stabilize on-policy learning when visual evidence is severely limited, we further introduce a budget-level curriculum that progressively decreases the token budget during training. Across nine benchmarks on Qwen3.5-4B, LT-OPD raises average retained performance under 5% visual-token retention from 68.6% to 82.3%, outperforming training-free, training-based, and reinforcement-learning baselines at the same budget. The gains transfer consistently to Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B. LT-OPD also reduces KV-cache usage by 85.2% and prefill FLOPs by 85.4% without additional inference overhead, demonstrating that on-policy learning can substantially recover capabilities lost to extreme visual-token reduction.
The Brief
2:52A short podcast summary of the paper. Written and voiced by AI.
Transcript
Fewer Tokens, More Self-Teaching. On-Policy Self-Distillation for Extreme Visual Token Reduction. By Junxian Li and five others, from Shanghai Jiao Tong University, Xi'an Jiaotong University, and two other institutions. Published on the twenty-sixth of September. Its code repository is worked out: about one hundred and eighty source files, with a readme and a licence.
Cutting down the number of visual tokens is an effective way to speed up a multimodal large language model. The catch is that performance falls away quickly once the token budget gets extremely low. Earlier work has attacked this from two sides: choosing which visual tokens to keep, and training the model to adapt to reduced visual input. The authors take a step further and ask a different question. When a model is heavily compressed, how should it learn from the states its own generations lead it into?
Posed that way, the setting calls for on-policy self-distillation. The compressed model is the student, and it is supervised on the trajectories it produces itself. Its teacher is the same model with every visual token intact, which makes it a rich source of information about what the student is missing. Two things are paired here. The states come from the compressed model's own generations, and the judgement on those states comes from the uncompressed one. The student is corrected on the paths it actually takes, by the version of itself that could see everything. LT-OPD is the training framework for extreme visual-token reduction built on this idea.
In practice, the student rolls out responses while seeing only a small fraction of the visual tokens. A frozen full-token copy of the same model then supplies distributional supervision along those student-generated trajectories. Learning on-policy is unstable when the visual evidence is this scarce, so LT-OPD adds a curriculum over the budget. Training starts with more tokens and progressively lowers the token budget as it goes, so the student is eased toward the extreme setting rather than dropped into it.
Across nine benchmarks on Qwen3.5-4B, with only five percent of visual tokens retained, LT-OPD raises average retained performance from sixty-eight point six percent to eighty-two point three percent. That beats training-free, training-based and reinforcement-learning baselines at the same budget.
The gains carry over consistently to Qwen3.5-9B, GLM-4.6V-9B and LLaVA-OV-1.5-4B. On the efficiency side, LT-OPD reduces key-value cache use by eighty-five point two percent and prefill FLOPs by eighty-five point four percent, and adds no inference overhead of its own. The student at test time is simply the compressed model, with nothing extra attached, so the savings from dropping tokens are kept in full.
The teacher in this setup is not a larger model or an outside one. It is the student's own uncompressed self, watching the compressed version work and correcting it on the paths it actually takes. The authors take the recovery they see as a sign that much of what extreme token reduction takes away can be learned back.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Fewer image tokens, faster model, falling accuracy. That is the trade this quest starts from.
You need a multimodal language model to run on tight hardware, so you keep only a tiny fraction of its visual tokens, the pieces of the image it reads.
The road ahead
- 1 · The problem ~7 min When a vision-language model keeps only a tiny fraction of its image tokens, choosing tokens better or copying reference answers no longer rescues it.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Let the compressed model write its own answers, have its full-sight twin grade every word, and shrink the token budget gradually.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Begin at a bigger token budget, let the student roll out answers, have the frozen full-token copy score them, and cut the budget step by step.
- 6 · The experiment ~6 min You’ll guess: Keeping only 5% of visual tokens on Qwen3.5-4B, what average share of its full performance did the model retain after LT-OPD training?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.