VA-TDM: A Unified View for Improving Few-Step Diffusion Models by Velocity Adversarial Distillation

VA-TDM. A Unified View for Improving Few-Step Diffusion Models by Velocity Adversarial Distillation. By Yihong Luo and two others, from the Hong Kong University of Science and Technology and the Chinese University of Hong Kong, Shenzhen. Published on the fourth of November.

Few-step distillation makes generation with diffusion and flow models efficient, because sampling takes only a handful of steps. Preserving quality at so few steps remains the hard part. Distribution-matching methods built on score distillation provide a strong foundation, but they rely on accurately tracking the student's distribution as it changes during training. At the intermediate steps of few-step sampling, estimating the student's score can be difficult. Worse, commonly used estimators may describe a different distribution from the one actually used to update the student. The feedback, in other words, can be about the wrong thing.

Flow matching opens a more flexible route to supervision: velocity fields constructed from the student's own sampling process. Building on that perspective, the authors introduce Velocity Adversarial Distillation, or VAD. A learned critic compares the student's field with the teacher's, and turns their disagreement into feedback for learning.

VAD is also offered as the unified view of the title. Score distillation, Fisher-based matching and moment matching are three named approaches, and all three fall within it. It also explains the existing MMD updates, which the abstract calls heuristic, so a frame built for one purpose ends up accounting for updates arrived at another way. And it does more than reorganise what exists: it delivers new training objectives as well.

VA-TDM builds on VAD. It follows trajectory distribution matching, which provides supervision for partial denoising along the student's trajectory. On top of that, it combines two further ingredients: transition-aware online velocity learning, and dual-form VAD supervision applied across the student's intermediate transitions. Those intermediate steps are exactly where the problem said student-score estimation was difficult, and they are where this supervision is aimed.

Extensive results show VA-TDM is powerful for learning few-step generators. It achieves state-of-the-art four-step performance in both text-to-image and text-to-video generation, across various backbones. Four steps is the budget the whole problem was about: few enough to be efficient, and few enough that quality is hard to hold on to. It also scales effectively to MiniMax-H3, a model of thirty-three billion parameters. In particular, VA-TDM-H3 notably outperforms the few-step Fast-H3, and according to human preference it is closer to the original many-step H3.

Velocity is the common thread through both halves. On the conceptual side, a critic reading velocity fields gives several existing distillation methods, and some heuristic updates, a single frame. On the practical side, the same signal drives a four-step generator that, at thirty-three billion parameters, moves closer to its many-step original.