ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation

ViRDM. Taming Representation Distribution Matching for Few-Step Causal Video Generation. By Zichong Meng and four others, from Northeastern University and Adobe Research. Published on the twenty-fourth of September. Its code repository is worked out: about eighty source files, with a readme and a licence.

Few-step autoregressive video diffusion makes low-latency streaming generation possible. Post-training for these models has mostly relied on Distribution Matching Distillation, or DMD, and DMD brings two more networks along with the generator. One is a large pretrained teacher. The other is an online critic, which estimates distributional discrepancies through diffusion scores. That is a three-network stack, and a resource-intensive one. Low latency when generating, in other words, has been paid for with a heavy stack when training. The paper asks whether the teacher and the critic can be removed entirely, by post-training only the generator against a target distribution computed in advance.

The starting point is representation distribution matching, or RDM, which comes from one-step image generation. The authors systematically study what happens when it is carried over to few-step causal video, and they identify three barriers. The gradient path becomes intractable in memory. Video has an optimization regime of its own, distinct from the image case. And representation distributions, by themselves, underconstrain temporal dynamics. Each of the three is a reason the image method does not simply transfer, and each needs an answer of its own. ViRDM is a post-training recipe, free of both teacher and critic, that takes on these barriers one after another.

The memory barrier is tackled first. ViRDM couples RDM with stochastically truncated clean-exit supervision, a lightweight VAE decoder, and staged vector-Jacobian products. Together these make representation distribution matching memory-feasible across multi-step causal video rollouts. For the second barrier, the authors establish which generated-population and initialization regimes are effective for video RDM. For the third, a lightweight dynamics regularization compensates for the temporal dynamics that representations alone leave underconstrained. The recipe works on memory, then on the optimization regime, then on time, in the same order the barriers were named.

The outcome turns three-network distillation into post-training of the generator alone. GPU memory use goes down, training time goes down, and video quality goes up. With only twenty generator updates, the recipe reaches eighty-four point eight seven on the official VBench evaluation. That is zero point three six above the previous best few-step causal baseline, and it requires sixteen A100 GPU-hours.

The authors also report exploratory results. They indicate potential for the same recipe at a lower causal sampling budget, and for bidirectional generation with one, two, or four steps. The abstract presents these as a demonstration of potential, set apart from the headline result.

Where DMD trains against a teacher and a critic, ViRDM trains against a distribution fixed before training begins. Each obstacle to making that switch in video receives its own targeted fix. The teacher and the critic are not made smaller or cheaper. They are removed. What remains at the end is a single network being updated, twenty times, to a VBench score above the best few-step causal baseline before it.