Programmable World Model

Programmable World Model. By Zheng-Hui Huang and ten others, from Alaya Lab. Published on the ninth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Video world models now generate visual experiences that are increasingly realistic and interactive. What they lack is a reliable way to keep a persistent world state, and to enforce rules that can be programmed, across long interactions. A world drawn frame by frame has nowhere to store what is true about it, so an object off screen or a character's hidden attribute has no guaranteed existence.

Programmable World Model separates two things video world models usually fuse: how the state of the world evolves, and how observations of it are generated visually. State is kept explicitly and updated by programs. Rendering is left to a video model. The world's facts live in one place and its appearance is produced in another, so the facts no longer depend on what the video model happens to draw.

An agent translates natural-language instructions into executable programs. Those programs specify the states of entities and the rules by which states change, giving direct control over individual entities and their interactions. A lightweight engine runs the programs and maintains an explicit, persistent global world state, one that includes entities off screen and attributes that are not visual at all.

To connect state to pictures, the authors introduce state-augmented 3D oriented bounding boxes, OBBs, as an intermediate representation. Those boxes, together with the target camera trajectory, are compiled deterministically into conditioning signals that are pixel-aligned in space and time. A pretrained video model receives those signals and acts as the generative renderer. Each step in that chain has a job. The program says what should happen, the engine records that it did, the boxes say where things are and what state they are in, and the video model turns all of that into frames.

The design lets users create playable games with predefined mechanics, direct control of individual entities, and a world state that persists throughout play. An entity can leave the view and return with its state intact, because its state never lived in the pixels.

To evaluate systems of this kind, the authors also introduce CombatStateBench. On it the method reaches ninety-four percent Count Accuracy and ninety-eight percent State Accuracy, substantially ahead of existing interactive video world models, while keeping generation coherent over long horizons.

The principle is a division of labour. Let an engine keep the books, and let the video model paint. The authors read the results as showing that separating explicit state evolution from generative rendering is an effective way to build worlds that persist and can be programmed.