World In-Context Learning for Agentic Robot Manipulation

World In-Context Learning for Agentic Robot Manipulation. By Heng Fang and five others, from Huazhong University of Science and Technology and ArcheOne AI. Published on the ninth of November.

A trained action policy writes its manipulation skill into network weights, and that ties what it learned to the distribution it learned from. In-context learning avoids the coupling, because nothing is written into the weights at all. Yet it has not matched trained policies, and the authors give a reason. A demonstration recorded in some other scene carries no feedback from the world the robot is actually facing now. It shows what was done somewhere else, but says nothing about whether doing it here will work. Plain in-context learning, working from a demonstration recorded elsewhere, never hears back from the scene in front of it.

The key observation is about what an agent learns in context. It is semantic: the steps of a task and the roles objects play in it, not their visual appearance. That kind of knowledge travels, so experience gathered in one world can transfer to another. From this comes World In-Context Learning, or WICL. A frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback from that rehearsal into its context instead of its weights. The demonstration gives a starting point, the rehearsal world gives the feedback that was missing, and the context is where the result is kept. Because what is kept is semantic, it does not stay tied to the simulator where it was learned.

The authors implement the paradigm as RoboScript, a modular infrastructure for agentic manipulation. Starting from a single demonstration, RoboScript learns each task by rehearsing it in simulation. The knowledge it gains is then frozen as a Task Memory. That frozen memory is deployed on new scenes, under a data isolation rule that guarantees every input used at deployment is one a real robot would also have. The sequence is learn, freeze, deploy. Rehearsal happens only in simulation, the memory stops changing once it is frozen, and the isolation keeps deployment honest: nothing reaches the deployed system that a physical robot could not also supply.

The test is all fifty tasks of RoboTwin 2.0. Across clean and randomized settings, with one demonstration per task and no gradient updates, RoboScript reaches eighty-three point four percent mean success. That is more than twenty points above the strongest trained action policy. The comparison is lopsided in setup: one side was trained, the other learned from a single demonstration plus rehearsal, with its weights never touched.

Several further properties are reported. The method generalizes across visual distributions. It improves through reflection. It scales with the foundation model underneath it. And it carries over to a physical dual-arm robot, through what the authors call cross-embodiment memory adaptation.

Where skill is stored has usually decided how far it travels. Weights hold it tightly and bind it to their training data. A context that has been through rehearsal holds it as steps and roles, which move between scenes. The paper's own summary puts it as a symmetry: with a world in the loop, a robot can learn a manipulation task in its context as well as it learns one in its weights.