Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

Hierarchical Residuals. A Minimalist Decoupling of Residual Width from Compute Width. By Ziteng Wang and six others, from Tsinghua University and ByteDance Seed. Published on the thirtieth of October.

Residual connections underpin Transformers, and they carry an implicit coupling. The width of the residual stream that travels through depth is tied to the compute width used inside each layer. Make one wider and the other comes along with it. Those are two different quantities doing two different jobs. One is the state the network carries from layer to layer through its depth. The other is the width at which each layer actually does its work. The standard design never lets them differ, and that coupling is implicit rather than a choice anyone makes.

Hierarchical Residuals, or HiRes, is a minimalist architectural change that breaks that tie. It widens the residual state while keeping the compute width of each layer fixed, so the two become separate dimensions that can be chosen independently.

The change works by grouping layers into blocks. Inside each block, standard PreNorm layers operate at the ordinary width, called d. Between blocks runs a wider outer residual of width D, and that outer path uses PostNorm. Learned read and write projections bridge the two spaces at the block boundaries, and the individual Transformer layers themselves are left unchanged. The whole construction is built from dense projections and normalizations only, so it needs neither specialized kernels nor infrastructure-level optimizations.

Across different model sizes and depths, HiRes consistently outperforms vanilla residual addition. Fitted scaling laws indicate a one point three-fold improvement in FLOPs efficiency. In a larger test, pretraining OLMoE-1B-7B on five hundred billion tokens, HiRes reaches a training loss lower by zero point zero two zero eight, and downstream accuracy higher by one point one four points.

Against other residual expansion methods, named as VWN, mHC and AttnRes, HiRes is competitive, and it gets there with less activation memory and lower input-output costs.

What the paper offers is less a new layer than a new knob. Residual width had been fixed by whatever the layers computed at. Separating the two, with nothing beyond standard operators, turns residual width into a practical scaling variable of its own. And because each Transformer layer inside a block is left as it was, the change sits around the existing layers rather than inside them.