Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Matrix-Game 4.0. Harnessing Video Models for Real-Time Playable Game Worlds. By Jihai Zhang and seventeen others, from Skywork AI, the Chinese University of Hong Kong, and five other institutions. Published on the twentieth of October. No code repository is linked to it.

A video model that responds to controls is not yet a game. To get from controllable video to a world you can genuinely play, the authors name three requirements: a world state that persists, game rules that actually execute, and interaction with low latency. A generator that only paints the next frames supplies none of those on its own.

Their answer is a roadmap for the next generation of Matrix-Game, built on one separation. Game execution and visual generation are pulled apart, joined through what they call an executable game harness. Matrix-Game four point oh is presented as the first system built on that design.

Three parts divide the work. A lightweight game engine holds the world state and executes the interaction rules. A multimodal coding agent takes a player's ideas and turns them into game code and into dynamic scene captions. And a renderer, based on an autoregressive video model, takes the engine's low-poly proxy frames and transforms them into detailed visuals, conditioned on those dynamic captions and on memory references. Because the engine is always producing something to render, the visual generator gets continuous guidance, and scenes and gameplay can be created or modified in natural language.

The renderer is trained on pairs of aligned low-poly and high-fidelity video, then distilled into a causal generator that works in chunks of four frames, using one or two denoising steps.

With targeted inference optimizations on top, the system runs gameplay at four hundred eighty p and sixty frames per second on a single NVIDIA RTX5090. The latency from a key press to the display is one hundred twelve milliseconds. Single-step generation reaches a steady-state throughput of eighty-three frames per second, with peak GPU memory of eighteen point two gibibytes.

Smaller cards were also tested with fixed prompts. Steady-state throughput reaches sixty-seven frames per second on an RTX5080, and fifty-eight on an RTX5070Ti. The authors demonstrate customizable shooter and driving games whose mechanics are enforced by the engine, and they release code, model weights and low-poly game assets.

Here the video model is no longer asked to be the game. The rules live in an engine that can be inspected and edited, the agent writes the code, and the generator's job narrows to making a crude proxy look real, fast enough to play.