Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

Mask Forcing. Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout. By Zhuoran Zhao and nine others, from the Hong Kong University of Science and Technology, Guangzhou, the Hong Kong University of Science and Technology, and four other institutions. Published on the eighth of September. Its code repository is public and real: about forty source files.

Autoregressive video diffusion models, generating video piece by piece, show great potential for real-time video generation. Recent methods obtain them by distilling a pretrained bidirectional video diffusion model into a causal autoregressive student, using Distribution Matching Distillation, or DMD. The videos that result often suffer from over-saturation and over-smoothing, which limits their visual quality and realism. The student is fast, since it generates causally, but it loses part of what made the teacher's output look real.

The authors trace a key cause to the objective. DMD uses a reverse KL objective, and reverse KL is mode-seeking. The student distribution can collapse onto only a few of the modes in the teacher's distribution, so a large part of what the teacher could produce is never learned. The saturated, smoothed look of the videos is the visible symptom of that narrowing.

Mask Forcing is a Dual-Noise Masking Rollout strategy. It perturbs the student's own rollouts during distillation, to counter the mode collapse that reverse-KL mode seeking induces. The core idea is to inject cleaner signals into the noisy rollout inputs, using random masks along both spatial and temporal axes, during the self-rollout process of autoregressive diffusion distillation. The masks are random, so where the cleaner signal lands varies across space and across time from one rollout to the next.

Those perturbations do two things. First, they push the student's rollouts to explore more regions of the teacher's distribution. That lets DMD supply learning signals beyond the modes the student already covers, which is exactly where mode collapse had left it blind. Second, the cleaner tokens act as denoising guidance for the noisier tokens around them. That improves the intermediate rollout predictions and reduces the accumulation of error as generation proceeds. In an autoregressive model each step builds on the last, so a better intermediate prediction is a better foundation for everything that follows.

Extensive experiments show the method improving several different autoregressive video diffusion distillation methods, raising visual quality efficiently. It does so without adding any real video data, and without extra post-training stages.

Only the rollout is touched. The objective stays, the teacher stays, the data stays. Masking some noise away changes what the student sees while it rolls out, and that is enough to widen what it learns from the teacher and to steady its predictions along the way. Because it attaches to the rollout, it can be applied to existing distillation methods rather than replacing them.