HiRAE: Hierarchical Representation Autoencoding with Residual Budgets
Xuanyu Zhu, Yan Bai, Yang Shi, Yihang Lou, Yuanxing Zhang, Tengfei Liu, Jing Jin, Yuan Zhou
Peking University · Agibot Research · Tsinghua University · IGDL
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 22 · arXiv 2609.37775
The abstract
Pretrained visual representations support image generation, but may not fully preserve the fine-grained details needed for faithful reconstruction. Meanwhile, intermediate encoder layers contain complementary visual details, but learning to fuse them for reconstruction can produce a latent distribution that is difficult to model. Existing fusion methods require empirical tuning of layer selection or staged optimization of fusion and decoding, increasing configuration effort or training complexity. We introduce HiRAE (Hierarchical Representation Autoencoder), which learns a hierarchical fusion framework over the full encoder hierarchy to improve reconstruction fidelity while maintaining compatibility with generative modeling. HiRAE groups encoder layers by depth and learns residual corrections to the deepest representation. Group-wise norm caps bound these corrections relative to the deep anchor, with tighter budgets for shallower groups. Our HiRAE-24 preserves the latent token count and channel dimension. On ImageNet-256, HiRAE-24 reduces reconstruction FID from 0.299 to 0.209 relative to RAEv2 while maintaining competitive guided generation quality. For text-to-image generation, HiRAE-24 improves alignment over RAEv2 on GenEval, DPG-Bench, and GenAI-Bench both before and after supervised fine-tuning. Under the same generator-training and evaluation protocol, post-fine-tuning GenEval increases from 84.86 to 87.70.
The Brief
2:26A short podcast summary of the paper. Written and voiced by AI.
Transcript
HiRAE. Hierarchical Representation Autoencoding with Residual Budgets. By Xuanyu Zhu and seven others, from Peking University, Agibot Research, and two other institutions. Published on the twenty-ninth of September. No code repository is linked to it.
Pretrained visual representations can support image generation, but they may not keep the fine-grained details that a faithful reconstruction needs. Those details are not lost entirely. Intermediate layers of the encoder hold complementary visual information. The trouble is that learning to fuse them for reconstruction can produce a latent distribution that is hard for a generator to model. Existing fusion methods cope by tuning which layers to use empirically, or by optimising fusion and decoding in separate stages. Either way, configuration effort or training complexity goes up.
HiRAE, the Hierarchical Representation Autoencoder, learns one fusion framework over the full encoder hierarchy instead of a hand-picked subset. Its aim is two things at once: better reconstruction fidelity, and continued compatibility with generative modelling. The deepest representation is kept as an anchor, and the shallower layers are only allowed to correct it.
Encoder layers are grouped by depth, and each group learns a residual correction to the deepest representation. Those corrections are held on a budget. A group-wise norm cap bounds each correction relative to the deep anchor, and the budgets get tighter for shallower groups. The shallowest layers can therefore contribute detail, but they cannot pull the latent far from where the deep representation put it. The version tested, HiRAE-24, keeps the latent token count and the channel dimension unchanged.
On ImageNet-256, compared with RAEv2, HiRAE-24 lowers reconstruction FID from zero point two nine nine to zero point two zero nine, while keeping guided generation quality competitive. For text-to-image generation, it improves alignment over RAEv2 on GenEval, DPG-Bench and GenAI-Bench, both before and after supervised fine-tuning. Under the same generator-training and evaluation protocol, GenEval after fine-tuning rises from eighty-four point eight six to eighty-seven point seven.
The difficulty named at the outset was never fusing layers as such. It was keeping the fused latent easy to model, and HiRAE's answer is a budget: each shallower group gets a smaller allowance for how far it may move the deep anchor. Detail comes back in, while the latent keeps its token count and its width.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A frozen encoder gives a great latent for generation, and a blurry one for reconstruction.
You generate images in the latent space of a frozen pretrained vision encoder. Its final layer is semantically rich, but reconstructions lose fine detail.
The road ahead
- 1 · The problem ~7 min A frozen encoder's final layer is good to generate in but blurs fine detail, and fusing in earlier layers can make the latent hard to generate.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Keep the deepest layer as an anchor, and let groups of shallower layers add only small residual corrections, capped tighter the shallower they are.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Group the frozen encoder's layers by depth, learn and cap each group's correction, add them to the deepest feature, then train a generator on the result.
- 6 · The experiment ~6 min You’ll guess: On ImageNet-256, fusing all encoder layers with capped corrections was compared with RAEv2. What happened to reconstruction and to guided generation?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.