Persistence Forcing: Exploiting Feature Specialization in Pixel-Space Diffusion

Persistence Forcing. Exploiting Feature Specialization in Pixel-Space Diffusion. By Chong Wang and five others, from Nanyang Technological University, the KTH Royal Institute of Technology, and one other institution. Published on the twenty-eighth of September. Its code repository is public and real: ten source files.

Pixel-space diffusion Transformers, DiTs, work directly on high-dimensional visual data, and their hidden representations are typically refined uniformly across depth. Every feature gets the same treatment at every layer. Natural images are not uniform in that way. They are organised at different levels of granularity. Global structure can often be represented compactly, while local textures and fine details need richer representations. A model that refines everything equally is not matching its effort to the structure of what it draws.

The authors introduce heterogeneous refinement. Different groups of features get different refinement budgets across depth. What comes out of this is an ordered specialization that was not imposed directly. Features refined sparsely end up encoding mostly global visual structure. Features refined more often increasingly specialize in localized, high-frequency detail. The first group is called persistent, the second active. Nobody assigned those roles. Give a set of features less refinement, and in the paper's account that is the job it settles into, while the heavily refined features drift toward fine detail.

Persistence Forcing, PerF, builds on that emergent split. It lets the persistent features continuously condition the active ones, so stable global information guides the ongoing refinement of finer detail. The outline steers the texture, rather than the two being worked out separately. During sampling, this interaction also produces a useful guidance direction of its own, one that promotes coherent global structure and naturally complements classifier-free guidance.

On ImageNet at two hundred fifty-six by two hundred fifty-six, PerF-L reaches an FID of one point nine one. That approaches the one point eight six of JiT-H, with only half the parameters. Lower is better on this measure, so the smaller model lands just behind a model twice its size. The larger PerF-H reaches an FID of one point six three at that resolution, and one point seven six at five hundred twelve by five hundred twelve.

One change in bookkeeping, giving features unequal refinement budgets, produces a division of labour on its own: some features hold the big picture, others chase detail. PerF's contribution is to wire that division together, so the part that settles early keeps guiding the part that is still being drawn.