The Past Frames the Future: Memory for Autoregressive Video Generation

The Past Frames the Future. Memory for Autoregressive Video Generation. By Harold Haodong Chen and twenty-four others, from the Hong Kong University of Science and Technology, City University of Hong Kong, and fourteen other institutions. Published on the twenty-third of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Generative models have improved video fidelity enough to support long-horizon generation, interactive world modeling, and visual environments that evolve. Autoregressive video generation extends a visual sequence through causal rollouts. But as the sequence grows, practical models must work under strictly bounded context windows, storage, and computation. Critical history, such as entity identities, dynamic states, and causal changes induced by interventions, often leaves the active context long before its relevance fades. Maintaining temporal persistence under those limits is a fundamental memory problem. A character who walks out of frame should look the same on returning, and a door that was opened should stay open, long after the frames showing it are gone from the model's view.

This review surveys memory mechanisms in autoregressive video generation systematically. It defines memory operationally: persistent historical information maintained across the outer autoregressive steps, able to influence future generation even after the evidence it came from is no longer locally accessible. The definition is about effect, not storage. Something counts as memory if it still shapes what comes next once the frames it came from have dropped out of reach.

On that unified footing, the literature is organized through five complementary perspectives. Forms are the representational carriers of history. Functions are the specific semantic and physical information that needs preserving. Operations cover the lifecycle of memory: writing, reading, updating, managing and integrating it. Learning concerns how memory behaviors are optimized under closed-loop rollouts. Evaluation covers the paradigms for diagnosing whether a memory capability is genuine. The first two ask what memory is and what it is for. The third asks what is done with it. The last two ask how it is trained and how anyone can tell it works.

The open challenges the review synthesizes are four: memory architectures that are composable and resource-aware, trustworthy updating of state, learning from a model's own rollouts, and standardized evaluation. Running beneath all of them is the operational definition. Memory is judged by whether history still shapes what is generated after the original evidence has left the window, and each perspective, from the form memory takes to the way it is evaluated, is a different angle on that one question. The challenges line up with the perspectives. Architecture speaks to forms, trustworthy updating to operations, self-rollout learning to learning, and standardized evaluation to evaluation.

By bridging representations, mechanisms and learning paradigms, the review aims to lay a structured foundation for video generation systems that are reliable and conditioned on memory. The title puts the stakes simply: the past frames the future, but only if something keeps the past.