Consistent Distribution Matching for Data-Free Diffusion Distillation

Consistent Distribution Matching for Data-Free Diffusion Distillation. By Yuxiang Fu and six others, from the University of British Columbia and the Vector Institute for AI. Published on the sixth of October. Its code repository is public and real: about twenty source files.

Flow and diffusion models are slow at inference, because generating a sample means running an expensive numerical integration. Distillation offers a way out: a student model learns from the teacher's dynamics, and can then generate in one step or a few. The trouble is what existing distillation methods tend to need along the way. Curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks. Each of these complicates training, and complicates scaling. A dataset has to be assembled and curated before distillation can start. Teacher rollouts mean running the slow integration many times over, which is the very cost distillation was meant to remove. And a proxy network is one more model to train and keep in step with the others.

Consistent Distribution Matching is a distillation method that is both simulation-free and data-free, aimed at accelerating diffusion and flow models while keeping their generative capacity strong. The key insight is to have one student network do two jobs that are usually separate: generating samples, and estimating scores.

Because a single student carries both roles, the framework needs only two models in total. One is a frozen teacher, the other a trainable student, and training optimizes a single objective. On the theory side, the authors prove that minimizing this objective indicates Wasserstein convergence of the student's flow-map pushforwards to the teacher's marginals. In plainer terms, driving the objective down is tied, formally, to the distributions the student produces approaching the ones the teacher defines. The abstract states that guarantee and does not go further into how the objective is built.

Results are reported on ImageNet at two hundred fifty-six resolution. With a single function evaluation, the method reaches an FID of two point zero four. With four function evaluations, the FID is one point three seven. Both come within forty epochs of training, and the authors report that these surpass the state-of-the-art distillation baselines that also work without data.

Much of the contribution is subtraction. No dataset, no teacher rollouts, no proxy network: a frozen teacher, a student, one objective, and a convergence guarantee tied to that objective. Code and model are released.