Learning Representations to Guide Pixel Diffusion Transformers

Learning Representations to Guide Pixel Diffusion Transformers. By Zihao Pan and eight others, from Sun Yat-sen University, Alibaba Group, and two other institutions. Published on the ninth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Visual representations have mostly been built for understanding images. What makes a representation useful for generating them is far less clear. The authors study that question in pixel diffusion transformers and come away with two insights. First, better representation quality does not automatically mean more use to generation: semantics aligned from outside the model can be progressively lost as pixels are synthesized. Second, separating the architecture, with specialized pixel decoders, mainly shifts where semantic and spatial modelling happen. It does not necessarily produce representations that suit generation any better.

From these, a requirement follows. A representation that helps generation has to be properly formed, and it has to be actively exploited. Representation Guidance, or RG, is built to do both directly, forming and using internal representations while keeping a plain Transformer architecture with no specialized decoder.

Two pieces carry it. A low-dimensional bottleneck forces the information that matters for generation into a compact representation. That representation is then used explicitly to guide what the decoder generates, by modifying the flow-matching target according to the prediction offset the representation induces. The authors report that RG learns clearer semantic and spatial structure this way, and that it scales efficiently in both parameters and computation.

On ImageNet at two hundred fifty-six by two hundred fifty-six, the RG-L/16 model, with only four hundred fifty-nine million parameters, outperforms pixel diffusion transformers that are substantially larger. The larger RG-XL/16 reaches an FID of one point five five.

At five hundred twelve by five hundred twelve, RG reaches an FID of one point seven four. It also extends to text-to-image generation, scoring zero point eight nine on GenEval and eighty-seven point one on DPG-Bench.

The lesson the paper draws is that a good representation is not enough on its own. A representation can score well for understanding and still fade away during pixel synthesis, and a more elaborate architecture can move the work around without improving it. What matters is that the representation is shaped for the job and then put to work. RG does the shaping with a bottleneck and the putting to work with a modified target. It keeps the plain Transformer while doing so, and the payoff shows up both in the class-conditional results, where a smaller model beats larger ones, and in text-to-image generation.