VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
Yuan Feng, Qize Yang, Ruizhe Chen, Sibo Song, Haolin He, Muzhi Zhu, Zihan Liu, Yunfei Chu and 4 others
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · The Chinese University of Hong Kong · Zhejiang University
Published October 6, 2026 · on the feed October 12, 2026 · ▲ 68 · arXiv 2610.07987
The abstract
Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.
The Brief
3:11A short podcast summary of the paper. Written and voiced by AI.
Transcript
VisionWeave. Weaving Elastic Visual Representations as a Native Capability of MLLMs. By Yuan Feng and eleven others, from the University of Science and Technology of China, Alibaba Token Hub, Alibaba Group, and two other institutions. Published on the sixth of October. Its code repository is public and real: about thirty-five source files.
Multimodal large language models dominate visual understanding, and they pay for it by encoding every input into dense patch tokens of a fixed size. But visual information is not spread evenly. Some regions need fine detail; others could be described compactly. Downsampling throws the detail away everywhere. Token pruning and other adaptive approaches exist, but they remain limited in how finely they adapt to content, in how well they generalize across tasks, and in how they fit with modern MLLMs and the infrastructure used to serve them. A fixed grid treats a blank wall and a page of small text the same way, and paying full price for both is where the cost comes from.
The authors argue that the fix belongs inside the foundation model itself. The model should learn, end to end, where to allocate visual representation and at what granularity. They call this native capability elastic visual representation weaving. VisionWeave sets out to establish it in frontier-level MLLMs through large-scale training. The word elastic is the point: the number of visual tokens is not set ahead of time, but stretches or shrinks with the content the model is looking at.
Two components do the work. A gated spatial pooler builds coarse-grained representations next to the native fine-grained ones, and both live in a shared MRoPE coordinate. A granularity router then learns how to allocate between them according to content. Coarse and fine are therefore not two separate pictures but two granularities placed on the same positional frame, and the router chooses between them. The training signal is self-distillation alone. The capability is first validated on Qwen3.5-4B, then scaled to Qwen3.8-27B, using over thirty thousand A100 GPU-hours.
On the larger model, VisionWeave adjusts its token savings to what the image contains. Across eight benchmarks it saves forty-three percent of tokens on average while keeping ninety-eight point nine percent of native performance. Token pruning baselines, held to a fixed savings target of fifty percent, preserve only eighty-eight percent. So the comparison is not like for like in savings, and the abstract does not claim it is: the point is that adaptive savings close to the same level lose far less.
Broader evaluations report robust trade-offs between efficiency and quality across diverse tasks, resolutions and video frames. Deployed on the SGLang serving engine, the method gives two point three times the throughput, cuts mean time to first token by fifty-four point four percent, and cuts mean time per output token by sixty point six percent.
The contrast with the baselines is in where the decision gets made. A fixed savings target decides in advance how much to cut from every input. Here the model decides per image how much detail each region deserves, which is how it can save less on one image and more on another, and still come out ahead on both cost and quality. The authors present elastic weaving as a capability for next-generation multimodal models.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a token bill that charges a blank wall as much as a page of text.
You run a multimodal model that reads images and video. Every input is cut into fixed-size patches, and every patch becomes a token, so high-resolution pages and long videos produce enormous sequences.
The road ahead
- 1 · The problem ~6 min Multimodal models pay the same token price for every patch of an image, so plain regions cost as much as dense text, and the usual fixes lose detail.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Give every image block two versions, four fine tokens or one coarse token, in one positional frame, and let a learned router pick per block.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Encode fine tokens, pool blocks into coarse tokens, place them at block centres, route each block, and train the whole thing by self-distillation.
- 6 · The experiment ~6 min You’ll guess: Across eight benchmarks on Qwen3.8-27B, what share of visual tokens did VisionWeave save on average?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.