Intrinsic Distribution Matching

Intrinsic Distribution Matching. By Deyuan Liu and nine others, from Inclusion AI, Westlake University, and four other institutions. Published on the twenty-ninth of October.

Text-to-image generation in one step, or in a few, has to render the details a prompt asks for, with very little opportunity to refine. Distribution matching distillation, or DMD, addresses this by using score differences to pull the generated distribution toward a reference distribution. Following prior work, the authors estimate that difference with internal velocity branches, which avoids training a separate fake-score network. Their question is whether an explicit moment constraint would improve this learned score supervision.

The answer starts from a decomposition. The reverse-KL field splits into a Gaussian moment component and a non-Gaussian residual. That split identifies statistics that can be constrained explicitly, rather than left entirely to the learned score. Put another way, part of what the score supervision is trying to correct can be named directly, as means and spreads, and those can be matched on purpose instead of only being reached indirectly.

Intrinsic Distribution Matching, or IDM, keeps the full DMD correction and adds to it. It matches feature means and per-coordinate standard deviations, and the features come from the generator's own exponential-moving-average copy. So no external feature encoder is required. Experiments cover both full-parameter tuning and low-rank adaptation, LoRA, on Z-Image-Turbo 6B and on Qwen-Image 20B.

At two network evaluations, IDM reaches ninety-four point two five percent on the Chinese LongText-Bench, and ninety point two percent on the English one. These are the highest scores among the methods compared at that budget. Controlled ablations attribute gains to the added moment loss. And features taken from the generator's moving-average copy perform competitively with an external DINOv2 encoder, across the five metrics reported.

The design adds a constraint without adding a component. The moment loss sits alongside the full DMD correction rather than replacing it, and the features it matches come from a copy of the generator already on hand. Nothing outside the model is consulted. The statistics being matched are intrinsic to the model being trained, and the ablations suggest that this does not cost much against borrowing features from an external encoder, while the explicit moment loss itself is where gains were measured.