Transferring the Intelligence of VLMs to Robotic Control

Transferring the Intelligence of VLMs to Robotic Control. By Meng-Hao Guo and eight others, from Tsinghua University and Tencent Hunyuan. Published on the nineteenth of September. No code repository is linked to it.

People move between physical and digital worlds without much trouble. There is a real gap between the two in embodiment, environment and task, yet human intelligence seems to carry across it. That observation prompts the question this paper asks: can the intelligence of vision-language models make the same crossing, from the digital world to the physical one, and control a robot?

The answer offered is yes, to a substantial degree, and without any task-specific robot training. A VLM given the right interface performs strongly on robot benchmarks zero-shot, and a single demonstration in context pushes it to state-of-the-art results.

The interface is called RoboDawn, and it is designed to be intuitive for a human. It exposes robotic control to an agentic VLM through a compact set of discrete commands: translation, rotation, and the gripper. The VLM runs in a closed loop. It observes the current visual state, reasons about the next action, executes it, and adapts what it does next to the state that results. On top of this, an in-context learning scheme, ICL, uses a few demonstrations to ground the model in two things: how to use the interface, and how to solve the task.

On RoboTwin two point zero C2R, the success rate is fifty-three point two percent zero-shot and seventy-three point six percent with one demonstration. The solid baseline there, pi zero point five, reaches forty-six percent. In the zero-shot setting, RoboDawn already outperforms several strong policies that were trained on robot data specific to the benchmark.

That whole gain comes from a single example shown in context, not from any task-specific robot training. On RoboDojo the pattern repeats. Success rises from thirty-five point six seven percent zero-shot to forty-seven point one seven percent with one demonstration.

The same framework also transfers to real robots, performing block-in-basket and block stacking on Franka.

The claim rests on two benchmarks, RoboTwin two point zero C2R and RoboDojo, plus real-world block-in-basket and block-stacking tasks on Franka. The control space is deliberately small: a compact set of discrete translation, rotation and gripper commands. The strongest results are the one-shot ones, which depend on a demonstration in context, while the zero-shot comparisons are against policies trained on benchmark-specific data. What is shown is transfer through this particular interface, not a VLM producing robot actions directly.

The finding turns the question of robot intelligence partly into a question of interface design. Given commands a human would find intuitive, and a loop that lets it look before each step, a model trained on the digital world already brings enough to control a robot. One example of how to use the controls, shown in context, is enough for a clear further jump.