Structured Residual Connectivity Matters for Diffusion Transformers

Structured Residual Connectivity Matters for Diffusion Transformers. By Yuhe Liu and four others, from the National University of Singapore and Shanghai Jiao Tong University. Published on the twenty-seventh of September. No code repository is linked to it.

Diffusion Transformers, or DiTs, have become a scalable backbone for high-fidelity image synthesis. Their wiring between layers, though, is plain. The U-Net based diffusion models that came before relied on skip connections that were rigid and designed by hand. DiTs mostly use a uniform residual stream instead, one that folds every preceding layer into a single monolithic state. Whatever an early layer computed reaches a later one only as part of that undifferentiated sum. The two families sit at opposite ends. One has structure that someone fixed in advance. The other has almost no structure at all, only accumulation.

The authors propose rethinking the residual connection itself: turning it from passive summation into an active retrieval mechanism, optimised for the job of denoising an image. The motivation comes from looking inside. A systematic analysis of the internal representations of a DiT reveals a latent preference for reusing early-layer features, along with a pattern of symmetric layer guidance. The network, left to itself, already leans toward certain cross-layer relationships that its plain residual stream does not make explicit.

Acting on that, they introduce a structured connectivity design that explicitly combines local residual connections with long-range pathways. It is positioned between two alternatives the abstract names: static skip connections on one side, and dense routing across all layers on the other. Here, each transformer block can selectively attend to the earlier representations that matter, retrieving spatial and semantic cues dynamically through direct, differentiable paths that cross depth. A block no longer has to take what the residual sum hands it. It can reach back for what it needs.

Experiments report faster convergence, needing up to one point seven three times fewer training iterations, together with significant gains in FID and in visual quality. The added parameters come to less than zero point one percent. Applied to a strong REPA-XL/2 model, the method improves FID without guidance from five point nine to four point three four, and reaches one point three nine FID with classifier-free guidance. The gains therefore arrive on two fronts at once. Training gets to its result in fewer iterations, and the result itself is better, on a model that was already strong before the change and for almost no extra capacity. Lower FID is the better direction throughout.

The target here is not anything inside a block. This work points at the connections between blocks, and argues from its results that adaptive cross-layer connectivity is a critical factor in diffusion transformers that has been underexplored. The price it reports is small, a fraction of a percent in parameters, and the authors present structured information pathways as a simple and effective direction for scalable generative models.