Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation

Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation. By Shukai Gong and eighteen others, from Peking University, AgiBot, and one other institution. Published on the first of October. No code repository is linked to it.

Real robot deployments increasingly ask for long-horizon compositional manipulation, where a single task is really several coordinated subtasks in sequence. World-action models, or WAMs, jointly predict short-horizon visual futures and the actions that go with them. What they typically lack is explicit reasoning at the level of subtasks. They see a little way ahead, but nothing in them says which piece of the larger job comes next.

ViGAR, short for Visual Goal-conditioned Action Reasoning, splits the problem in two. It is a hierarchical framework that factorizes manipulation into a visual subgoal planner and a subgoal executor. The planner decides what the next stage should look like. The executor works out how to get there.

Given the current observation and the global instruction, the planner predicts a visual subgoal for the next subtask: an image of where things should be. The executor then generates future visual trajectories and actions jointly, conditioned on that predicted subgoal. Both components share a pretrained world-model representation, so planning at the level of tasks and generating actions both draw on the same common physical knowledge. One more property falls out of the design. Because subgoals are visual, the framework supports in-context learning. Supplying a global goal image as context can induce different subtask decompositions and different behaviours, with no parameter updates at all.

On the RoboTwin Clean2Random benchmark, ViGAR reaches an eighty-two percent success rate in the clean setting. In the random setting it reaches sixty-seven point zero two percent. Averaged, that surpasses the strongest baseline by twelve point eight six percentage points. Real-world robot experiments cover five compositional tasks and two in-context learning tasks, and the authors report that these further confirm the framework's effectiveness.

What changes here is where the plan lives. Instead of a single model looking a short way ahead, ViGAR names the next subgoal as a picture and hands it to an executor, and that picture doubles as the handle for in-context control: show the robot a different goal image, and it breaks the task down differently.