From Pixels to Actions - Towards Native Vision-Language-Action Policies
Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang and 9 others
S-Lab, Nanyang Technological University · SenseTime Research · Ropedia
Published November 4, 2026 · on the feed November 5, 2026 · ▲ 62 · arXiv 2611.06918
The abstract
Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.
The Brief
2:16A short podcast summary of the paper. Written and voiced by AI.
Transcript
From Pixels to Actions - Towards Native Vision-Language-Action Policies. By Weichen Fan and sixteen others, from S-Lab at Nanyang Technological University, SenseTime Research, and one other institution. Published on the fourth of November.
Vision-language-action models are typically built on a simple premise. Perception is first compressed by a visual encoder, and action reasoning begins only afterwards. Manipulation, though, often hinges on precisely the visual signals most vulnerable to that abstraction: local geometry, boundaries, depth, and subtle changes of state. So a fundamental question follows. Should an action policy inherit a fixed visual endpoint, or reach directly into visual computation as it evolves? The claim is that it should reach in. Embodied control benefits from access to visual representations across the computational hierarchy, rather than from a fixed encoder output.
The vehicle for the claim is NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers, and they evolve jointly with language throughout the backbone. That lets a flow-matching action expert draw on instruction-conditioned visual representations at different depths.
This simple change reveals a consistent pattern. Under controlled comparisons, native architectures outperform their encoder-based counterparts. Full fine-tuning reduces the gap but does not erase it, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the signals critical for action often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control.
The evidence comes from manipulation, under controlled comparisons and distribution shifts, and the authors state their conclusion with care: the results suggest the benefit, rather than establish it in general. The probing result is hedged in the same way. Action-critical signals often peak early, which is not the same as always. And the comparison is architectural, native designs against encoder-based ones, with full fine-tuning and added scale tested as ways the encoder-based side might close the gap. Neither does.
A fixed encoder output is one endpoint, settled before action reasoning starts. The picture NEO-VLA offers instead is a policy that can draw on visual computation while it is still in progress, at whichever depth carries the signal it needs.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A robot has to see an edge before it can grip it. The usual recipe may be hiding that edge.
You are building a robot model that reads a camera image and an instruction, then moves a gripper. The usual recipe first squeezes each image through a pretrained visual encoder, and only its final output reaches the action part.
The road ahead
- 1 · The problem ~7 min Robot models usually let the action part see only a pretrained encoder's final image summary, which may drop the fine details grasping needs.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Remove the separate encoder so vision lives inside the backbone, and let the action expert read visual representations from several depths.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Pixels become tokens, share a backbone with words, and a flow-matching expert reads chosen layers to output continuous action chunks.
- 6 · The experiment ~6 min You’ll guess: When the encoder-based counterparts were fully fine-tuned, what happened to their gap behind NEO-VLA?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.