MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

MotorMind. Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation. By Bingxuan Li and five others, from the University of Illinois Urbana-Champaign. Published on the twenty-ninth of September. No code repository is linked to it.

Vision-language-action models have pushed robotic manipulation forward, but two things hold them back. Their zero-shot generalization to new tasks and new environments is still limited. And because they depend on specialized training, they cannot directly benefit from general-purpose vision-language models, which are advancing quickly.

A second line of work takes a different route. Agentic robotic systems use vision-language models for high-level reasoning, or use coding agents to control the robot. These often depend on many external models and tools, and every one of them adds complexity and cost.

That sets up the question the authors pose. Could a general-purpose vision-language model operate a robot by itself, more like a human teleoperator would? That means reasoning directly from what it observes, issuing actions, and continually adapting to feedback from execution. And it means doing so without learned action experts, without coding agents, and without grounding tools such as SAM3.

MotorMind is their answer: a robot manipulation harness built around the model rather than a new model.

The division of labour is the core of the design. The vision-language model proposes actions at a middle level of abstraction, and MotorMind connects those mid-level actions to deterministic robot control. Feedback from execution then flows back to the model, which is what lets it keep adapting, the way a teleoperator watches the arm and corrects. Two further pieces run alongside the main loop: monitoring that happens asynchronously, and memory that is updated in the background.

Just as telling is what is absent. There is no task-specific policy training anywhere in the loop, no coding agent writing control programs, and no extra grounding tool such as SAM3. The general-purpose model does the reasoning, and the harness does the rest.

On the base LIBERO-PRO suites, MotorMind reaches sixty-six point seven percent success. Under perturbations it reaches fifty-three point eight percent. The prior zero-shot methods the authors evaluate reach at most thirteen point three percent on the base suites, and at most nineteen point two percent under perturbations. On the base suites, that is roughly five times the best prior zero-shot result.

The same interface was also run on a real xArm6 robot, across direct manipulation and settings where a human perturbs the task, and averaged ninety-five percent success. Swapping in a stronger vision-language model as the backbone improves performance further. The failures that remain come mainly from visual grounding, embodied reasoning and action knowledge, and they shrink as the model's capability grows.

The conclusion the authors draw is a conditional one. A general-purpose vision-language model can do effective zero-shot manipulation, provided it is given the right mid-level action representation and an asynchronous execution harness to work through. The scaffolding is the contribution. And because the remaining failures decrease as the backbone gets stronger, the same harness stands to gain from progress in general-purpose models, which is exactly the benefit that specialized training had been keeping out of reach.