FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders

FuseReg. Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders. By Hongyang Du and fifteen others, from the University of Southern California, Brown University, and five other institutions. Published on the twenty-fifth of September. Its code repository is worked out: about seventy source files, with a readme and a licence.

Representation autoencoders, or RAEs, reuse features from a pretrained visual encoder as the latents for both reconstruction and diffusion, bringing strong visual representations into image generation. They still have to decide which encoder layers form the shared latent space that the generator and the pixel decoder both use, and that choice is a trade-off. Shallower layers tend to preserve fine pixel detail better. Deeper layers tend to yield better generation metrics. A fixed heuristic fusion of layers therefore couples two stages that benefit from different information.

FuseReg replaces heuristic feature selection with training over random subsets of encoder layers. Instead of committing to one fusion in advance, the downstream model is trained across many of them. The question of which layers to use stops being a design decision made once, by hand, and becomes something the downstream model is trained to handle whatever the answer turns out to be.

The authors analyze the mechanism theoretically. Subset sampling explicitly penalizes sensitivity to disagreement across layers. Put plainly, a model trained this way is pushed not to depend on the layers agreeing with one another. And all of this happens downstream: the pretrained encoder itself is not modified.

The experiments use ImageNet-256, with DINOv3-L as the encoder. A single FuseReg decoder reconstructs from full, sparse, and single-layer fusions without retraining, and it achieves higher PSNR than decoders specialized to fixed fusions. One decoder, in other words, does the work of several, and does it better than each of them on its own fusion, which is the reconstruction side of the trade-off addressed.

That flexibility also benefits generation. Replacing the decoder alone, with an unchanged RAEv2 DiT-XL generator, reduces unguided gFID by twenty-seven percent. The same regularization principle extends to diffusion training. Regularizing both stages jointly reduces unguided gFID by twenty-nine percent on DiT-Base. In the first of those numbers, notice what stays fixed: the generator is untouched, and only the decoder is swapped. The second shows the idea is not confined to the decoder, since the generator can be regularized the same way.

Training downstream models for robustness to layer fusion, the authors conclude, narrows the reconstruction-generation gap. The encoder stays as it was. What changes is that neither the decoder nor the generator is tied to a single hand-picked answer about which layers to use, and one decoder can serve where several specialized ones stood before.