World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

World Action Agent. Harnessing VLMs for Robot Manipulation via World Action Rehearsal. By Yehang Zhang and fifteen others, from the Hong Kong University of Science and Technology, Guangzhou, the Chinese University of Hong Kong, and one other institution. Published on the twenty-fourth of September. No code repository is linked to it.

General-purpose vision-language models bring broad knowledge and spatial reasoning to robot manipulation. Existing systems, though, tend to hold them at arm's length in one of two ways. Some use them indirectly, asking them to predict constraints or to write programs that something else carries out. Others give them a view of the scene, but not a world in which to act. Either way, the model's knowledge reaches the robot only secondhand.

World Action Agent, or WAA, is a multi-agent harness through which a vision-language model pilots a robot directly, using basic tools. Every decision is made inside a visual action workspace, and the workspace is the idea. It has three properties, each aimed at letting the model act where it can see.

Contact views come first. Selected automatically from the scene geometry, they present the scene around the current interaction. Action rehearsal comes second. Each action becomes an editable proposal, which the agent previews and revises against planning feedback before anything is executed, either on its own or through an Imagination Agent. In-view correction is the third. It closes the loop between observation, rehearsal, and low-level execution, so the agent can remove residual offsets in the very view where it observes them.

The same workspace is also how WAA learns procedure. It evolves multimodal skills from expert videos and from human teaching, under evidence-based review, and consults those skills through a Skill Agent. Its interaction traces, in turn, become training data for smaller vision-language models, which learn to pilot the same harness.

On LIBERO-Pro, WAA uses skills evolved only from LIBERO-90, and reaches an average success of seventy-five point six percent, which the authors report as state of the art. That beats end-to-end vision-language-action models, code-as-policy agents, and a visual-harness baseline built on the same backbone. The skills also carry over to robosuite with no further learning and remain effective there.

The traces pay off as well. Fine-tuning Qwen3.5-9B on harness traces lifts its out-of-domain success from one point seven percent to forty-three point three percent.

Where earlier systems had the model advise from outside, this one puts it in the driver's seat and gives it a place to try before it commits. Seeing the contact, rehearsing the action, and correcting in view are the three moves that make direct piloting workable. The skills it gathers and the traces it leaves behind then turn one agent's experience into something reusable, both for the next task and for a smaller model.