Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation
Chenjian Gao, Zhihao Hu, Jianqi Ma, Jun Zhang, Weidong Zhang, Tianfan Xue
MMLab, The Chinese University of Hong Kong · Tencent AIPD
Published September 29, 2026 · on the feed October 5, 2026 · ▲ 23 · arXiv 2609.37925
The abstract
Autoregressive (AR) video diffusion enables low-latency, streamable video generation, but prediction errors often accumulate over long rollouts. Training the generator on its own rollouts exposes it to these imperfect histories. However, existing video-level distribution matching distillation (DMD) scores the whole rollout jointly. Because a chunk is evaluated together with its past and future, its correction can favor matching artifacts in the surrounding context merely to preserve temporal consistency. To provide a clearer visual-quality signal, we introduce Rollout-Marginal Distillation (RMD). RMD retains the generated history for AR prediction but scores each chunk independently against a chunk teacher, ensuring its quality correction is not compromised by an imperfect temporal context. To compensate for the lack of temporal context in independent chunk scoring, RMD subsequently applies video-level DMD to restore temporal coherence. Extensive experiments demonstrate that RMD maintains high visual quality far beyond its training horizon and outperforms video-level DMD baselines. Code and video results are available at https://cjeen.github.io/RMD
The Brief
2:04A short podcast summary of the paper. Written and voiced by AI.
Transcript
Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation. By Chenjian Gao and five others, from MMLab at the Chinese University of Hong Kong and Tencent AIPD. Published on the twenty-ninth of September. Its code repository is worked out: about forty-five source files, with a readme and a licence.
Autoregressive video diffusion produces video one piece after another, which is what makes it low-latency and streamable. The price is that prediction errors pile up over a long rollout, each step building on whatever the steps before it got wrong. One response is to train the generator on its own rollouts, so that during training it is exposed to the same imperfect histories it will face in use.
The trouble lies in how those rollouts are scored. Existing video-level distribution matching distillation, DMD, scores the whole rollout jointly. A chunk is judged together with what came before it and what comes after it. That means the correction it receives can end up favoring a match to artifacts in the surrounding context, simply to keep the video temporally consistent with them. The flaw gets preserved because it is consistent.
Rollout-Marginal Distillation, RMD, pulls those two concerns apart to give a clearer signal about visual quality. The generated history is still kept and still used for autoregressive prediction, but each chunk is scored on its own, independently, against a chunk teacher. Its quality correction no longer has to answer to an imperfect temporal context.
Scoring chunks independently has a cost of its own: it throws away the temporal context that holds a video together. To make up for that, RMD then applies video-level DMD as a later step, to restore temporal coherence. Quality is corrected first in isolation, and coherence is put back afterwards, so each signal does the job it is suited to.
Extensive experiments show RMD keeping high visual quality far beyond the horizon it was trained on, and outperforming video-level DMD baselines.
The fix is a matter of ordering. Scored jointly, a chunk is pulled toward consistency with earlier mistakes. Scored alone first, then stitched back into a coherent whole, it is pulled toward looking right.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free