Show-Harness: Just a VLM Agent Can Play Robots
Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma and 2 others
National University of Singapore
Published September 9, 2026 · on the feed September 10, 2026 · ▲ 160 · arXiv 2609.10522
The abstract
Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.
The Brief
2:28A short podcast summary of the paper. Written and voiced by AI.
Transcript
Show-Harness. Just a VLM Agent Can Play Robots. By Yanzhe Chen and nine others, from the National University of Singapore. Published on the ninth of September. Its code repository is worked out: about two hundred source files, with a readme and a licence.
Foundation vision-language models, VLMs, show broad intelligence about the world. Turning that intelligence into control of a robot is still hard. The gap is between a model that reasons in meaning and a machine that needs precise physical commands.
Show-Harness is what the authors call an Embodied Harness. It lets a VLM play a robot, in their word, through a compact semantic interface that links intent to action. The title makes the claim directly: just a VLM agent can play robots. What links the VLM to the robot is the interface. The VLM chooses, and the interface carries the choice out.
The interface exposes discrete semantic action units, things a VLM can naturally reason about. Interpreters specific to each embodiment then ground those units deterministically into local robot actions. The division matters. Grounding is deterministic, so the VLM itself stays directly responsible for the fine-grained physical decisions, rather than handing them off.
The same interface serves two purposes. It can directly unlock closed-source frontier VLMs for zero-shot robot control. And it can adapt small open-source VLMs for low-cost deployment, with just a few GPU-hours of fine-tuning. One route takes the most capable models as they are. The other makes a small model good enough to run cheaply. Both depend on nothing more than the shared set of semantic actions.
The authors also build GUMI, the GUI Manipulation Interface. It extends the same semantic action space to collecting demonstrations through a GUI, so humans and agents can play robots across different embodiments without specialised teleoperation hardware. Because demonstrations are recorded in the same action units the VLM reasons over, what is collected this way speaks the same language as the model that will learn from it.
Extensive experiments show VLM agents equipped with Show-Harness generalizing robustly across tasks, embodiments and environments, and outperforming representative agentic and VLA paradigms. The abstract gives no specific figures.
In the authors' reading, the right interface can unlock substantial embodied capability from foundation VLMs without adding model capacity and without costly pretraining for each embodiment. The capability was already in the model. What was missing was a language of actions it could speak.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A model that understands the table still cannot move the arm. That is the wall.
You want a robot arm to tidy a table using a vision-language model that already knows objects, spatial relations and how to split long goals into parts.
The road ahead
- 1 · The problem ~6 min A vision-language model already understands a messy table, but neither usual way of wiring it to a robot arm keeps that understanding in charge of the motion.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Give the VLM a small vocabulary of one-step moves and gripper actions, and let fixed rules turn each chosen word into a small, bounded robot motion.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each step runs a loop: offer the VLM its vocabulary of action units, let it choose one, then have the robot's interpreter turn it into motion.
- 6 · The experiment ~6 min You’ll guess: Through the same semantic interface, how much fine-tuning did small open-source VLMs need to be adapted for low-cost robot control?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.