CtrlCache: Accelerating Interactive Video World Models with Control-Aware Caching

CtrlCache. Accelerating Interactive Video World Models with Control-Aware Caching. By Shangye Song and four others, from the University of Auckland and the University of New South Wales, Sydney. Published on the sixth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Interactive video world models have two jobs at once: generate each chunk of video efficiently, and respond faithfully to whatever the user is doing with the controls. Many systems generate chunk by chunk, autoregressively, with few-step denoising. Even so, each chunk still needs several costly denoising iterations.

Training-free caching can cut that cost by reusing computation. But existing caching policies decide what to reuse mainly from the model's internal denoising dynamics. They do not explicitly account for transitions in the controls. And interactive generation hands them a signal they leave unused: the controls for a chunk arrive before that chunk is denoised. A schedule built from the controls therefore costs no forward pass at all.

The authors first analyse adjacent chunks under different control regimes. Two things stand out. Structural similarity drops around action changes. And low-frequency structure persists longer than high-frequency detail. CtrlCache is built on both observations: a training-free, control-aware caching framework that adapts how much computation it spends to the current control sequence.

Its action-aware scheduling and refresh policy detects action changes, both across chunks and within them, and labels each chunk as one of four states: initial, transition, turning, or steady. At one selected interior denoising step, initial and transition chunks keep full computation. Turning and steady chunks instead reuse the transformer residual from the most recent fully computed step in the same chunk. The split follows the analysis directly. Around an action change, structure shifts, so the chunks where the controls have just changed are the ones that get spent on. Where the controls are holding steady, there is less new structure to compute, and reuse is allowed. All of this is decided from the control sequence, which is already known, so the decision itself is free.

The second component targets steady interaction, where low-frequency structure persists. A frequency-mixed history prior guidance brings in complementary information from the preceding clean latent, and it does so without an additional DiT forward pass. The cheap signal, the controls, decides where to save, and the persistent part of the picture is carried forward from what was already generated.

CtrlCache was evaluated on three models: Matrix-Game two point oh, and LingBot-World in its first and second versions. Without any model retraining, it achieves speedups of the DiT backbone ranging from one point two one times to one point four one times. At the same time it improves WBench Overall scores over the original inference, on all three models. So the speedup does not come at the cost of the benchmark score; the score goes up as well.

Caching decisions are usually made by watching the model from the inside. CtrlCache looks at something that arrives from outside and arrives early: the user's own controls. Because those are known before a chunk is denoised, they give a free schedule for where full computation is needed and where it can be reused, which is a natural fit for generation that is interactive by design.