WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

WorldGuide. Goal-Directed Video World Model for Procedural Task Execution. By Ankan Deria and four others, from the Mohamed bin Zayed University of Artificial Intelligence. Published on the eighth of October.

Video generators and video world models can already produce visual trajectories that look plausible. A long procedural task asks for something else. The generation has to adapt to what it has actually produced so far. At each point a model must work out the next action from the state it generated, carry that action out, and notice when the task is finished.

Open-loop generation cannot adapt to how execution went. Existing closed-loop systems often depend on pretrained executors, or on indirect verification. Either way, a gap remains between deciding on an action and successfully realizing it. Choosing the right next step is only half the job. The step then has to actually appear in the generated video, and the system has to see whether it did before it can decide what comes after.

WorldGuide treats procedural video generation as closed-loop task execution, carried out in visual world space. It is given only an initial image and a task goal. From there it predicts one atomic action, generates the video clip for that action, and then looks at the result to choose the next action, or to stop. Each clip becomes the starting state for the next decision. The loop runs on the model's own output rather than on a fixed script written in advance, which is what lets it respond to how each step actually turned out.

Two components share the work, a Planner and an Executor, and both are trained on the same step-level procedural demonstrations. The Planner learns to predict the next atomic action, or that the task is complete, from the visual progress so far. The Executor is trained directly to realize the actions the Planner predicts. Because the two learn from the same demonstrations, what one proposes is what the other was taught to carry out. Across a long execution, a hierarchical visual memory keeps track of state while holding the cost in history tokens bounded.

Training a planner and an executor jointly needs step-level supervision that pairs actions with video, and that did not exist. The authors therefore built WorldGuide Bench: about fifty-nine thousand step-annotated videos, covering two hundred and forty-five tasks in twenty-seven procedural categories.

On WorldGuide-Bench, WorldGuide reaches a Task Success of thirty-three point three three percent. The strong recent video model MiniMax-H3 reaches twenty-nine point nine percent, and it does so even though it is given reference action plans.

On VideoCraft-Bench, with both models conditioned only on the goal, WorldGuide reaches forty-seven point six nine percent. MiniMax-H3 reaches thirty-two point seven three percent. Under matched conditions, then, the margin is wider than on the first benchmark, where the comparison model had the extra help of a reference plan.

The comparison that carries the most weight is the first one. A model handed the plan still trails a model that has to work the plan out for itself, one step at a time, from what it has generated so far. The authors read this as evidence that planning and learned execution belong in the same loop for goal-directed procedural video generation.