On the Diffusibility of High-Dimensional Latents

On the Diffusibility of High-Dimensional Latents. By Chao Feng and nine others, from Cornell University, Adobe, and three other institutions. Published on the twenty-third of September. No code repository is linked to it.

Representation Autoencoders let diffusion models operate in the feature spaces of pretrained visual encoders. Many off-the-shelf encoders are not optimized for faithful reconstruction, and discard fine-grained visual detail. Finetuning them for image reconstruction recovers that detail, as expected. The counterintuitive part is what else it does. It reduces the effective dimensionality of the resulting representation, and that altered geometry has downstream effects on generation.

Specifically, the authors show that standard velocity prediction in flow matching, used in this high-dimensional space, requires the model to fit noise directions that are orthogonal to the low-dimensional signal manifold. That makes optimization inefficient. It motivates switching to the clean data parameterization, x zero prediction, which focuses learning on the underlying signal manifold. Across experiments with multiple strong-reconstruction encoders, x zero prediction consistently improves text-to-image generation.

The result is stated for representation autoencoders built on encoders with strong reconstruction, for flow matching, and for text-to-image generation. The contrast it draws is between two parameterizations, velocity prediction and clean data prediction, in a space that is high-dimensional while its signal lies on a lower-dimensional manifold.

A better encoder in the reconstruction sense is not automatically an easier space to diffuse in. When the signal occupies fewer dimensions than the space offers, predicting the clean data rather than the velocity keeps the model's effort on the directions that carry signal, instead of on the noise around them.