WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

WorldLine. Action-Driven Visual Simulation for Robotic Manipulation. By Shenghe Zheng and six others, from the Hong Kong University of Science and Technology, Joy Future Academy, and two other institutions. Published on the twenty-ninth of September. Its code repository is public but slight: three source files.

Learning robot behaviour in the real world is limited by cost: the cost of collecting experience, and the cost of evaluating candidate behaviours. A visual simulator that predicts what an action will do, before the robot physically does it, would ease both, and video generation models offer a scalable base to build one on. Two obstacles stand in the way. Video models tend to favour what looks plausible over following the action accurately, and over coherent dynamics between robot and object. Simulators conditioned on actions, meanwhile, depend on data that is scarce and specific to one embodiment, and hard to share between robots whose control spaces do not match.

WorldLine is an action-driven visual simulator that separates those two needs. Learning dynamics that transfer is one job. Grounding them in a particular robot's actions is another. The first can draw on large amounts of video with no actions attached; the second needs much less.

In numbers, WorldLine learns manipulation dynamics from more than ten thousand hours of action-free robot video, then grounds them with over two thousand hours of action trajectories spanning more than ten embodiments. A shared control interface across those embodiments comes from representing actions in image space. Training is multi-view and enriched with failures, and relational regularisation improves predictions that hinge on interaction. Finally, a few-step distillation focused on the robot makes causal rollouts efficient while preserving the motion that matters for the action.

In both held-out and out-of-domain settings, WorldLine keeps strong visual quality and agreement with the robot's motion. On failed trajectories, it improves robot-mask IoU over the strongest baseline by zero point one six two six. As a judge of outcomes, it predicts whether a trajectory will succeed with seventy-four percent mean accuracy across RoboTwin and AgiBot, one percentage point above the strongest baseline.

Last comes use in a loop with a policy. Without any training on RoboTwin or any adaptation to it, rollouts from WorldLine improve task success by up to twenty-one point four percentage points over executing the policy directly.

Video without actions is plentiful, and actions are expensive and tied to one robot. WorldLine is organised around that asymmetry: it takes its sense of dynamics from the cheap source and its sense of control from the scarce one. The authors present the result as a scalable and efficient simulator for evaluating policies and for planning with embodied agents.