ROWBench: Do Video Models Render What the Program Specifies?

ROWBench. Do Video Models Render What the Program Specifies?. By Zheng-Hui Huang and eight others, from Alaya Lab. Published on the first of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Programmable world models split executable dynamics apart from visual generation, which makes them a promising foundation for the next generation of game engines. How faithfully their visuals follow the explicit rules and interactions of the program has not been evaluated well. Existing benchmarks look at visual quality, controllability, and adherence to instructions or to physics. They rarely test whether a video matches fine-grained world events that a program specified. If the program says a door opens at a certain moment, existing benchmarks rarely check that the video shows the door opening then.

The benchmark, written PROWBench in the abstract, checks generated videos against the observable consequences of program execution. It evaluates entity control and long-horizon memory. Two further metrics, both based on vision language models, cover timing and events: Logic-Render Alignment measures adherence to the prescribed timeline, and Interaction Success Rate measures whether timestamped events recorded by the engine are actually realised on screen.

PROWBench holds one hundred and seventy programmatically constructed episodes and six hundred proxy videos, covering diverse scenes and interactions. Each episode logs entity states and timestamped events as a replayable world record, including events outside the camera's field of view. From those records it renders synchronized views and proxy representations. An extensible framework builds the scenes, controls behaviours, and can render any camera view in different forms, such as coarse 3D or bounding boxes. Both first-person and third-person perspectives are covered, and a subset of episodes adds synchronized multi-view observations.

The abstract reports no results for any model. It describes what the benchmark measures and how it is built, but not what it found when video models were put through it, so there are no scores to relay here. Whether current models render what the program specifies, the question in the title, is left for the paper itself to answer.

A world record logged by the engine works as a script the video is supposed to follow, down to events off camera. Grading a generated video against that script asks a narrower and stricter question than whether it looks good: did it show what the program said happened. Because the record also holds what happened out of view, even memory can be checked, against events the camera never saw.