DepthBench: Measuring How Residual Connections Enable More Computational Depth

DepthBench. Measuring How Residual Connections Enable More Computational Depth. By Keyu Wang and five others, from the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, and two other institutions. Published on the twenty-sixth of September. Its code repository is worked out: about four hundred and fifty source files, with a readme.

Adding depth is a natural way to give a Transformer more computational capacity, yet the deeper layers tend to contribute less and less as depth grows. Recent approaches try to fix this. Some strengthen normalization, as LayerNorm Scaling does. Others change the residual connections, as mHC and AttnRes do, aiming for better information flow and better use of depth. What has stayed unclear is whether these methods really turn extra architectural depth into effective computational depth, and whether their reported gains come from better access to information across layers or from confounding factors nobody accounted for.

DepthBench is a controlled benchmark for studying computational depth across architectures. The question it asks is simple to state: if you move capacity from width into depth, does the model get better, stay the same, or get worse? And does the answer depend on the architecture? The distinction it draws is between architectural depth and computational depth. The first is simply how many layers a model has. The second is how much of that stacking actually turns into useful computation. A model can be deep on paper and shallow in practice, if its later layers add little, and DepthBench is built to tell those two situations apart. It measures pre-training loss, but also domain-specific performance and how effectively the extra layers are used.

Control sits at the centre of the design. The authors systematically vary the ratio of model width to number of layers, sweeping from shallow and wide shapes to deep and narrow ones. Throughout, the total model size and the pre-training recipe are held fixed, so the only thing that changes is how the same capacity is arranged. Ten representative architectures are run through this sweep, which is what lets the benchmark separate the effect of depth from the confounding factors the gap described. If a method improves only when it is also given a different recipe or more parameters, this setup would not credit it.

The main result is that the benefit of putting more capacity into depth is strongly architecture-dependent. Standard Pre-LN, along with most of its variants based on normalization or scaling, gains little, and can even get worse as models become deeper and narrower. HC and Full AttnRes, by contrast, improve consistently, even at extreme deep shapes.

Those gains are not confined to pre-training loss. They translate consistently into better domain-specific performance and more effective computation. Controlled analyses at the level of individual layers go further, showing that the gains of HC and Full AttnRes come with more effective use of the additional layers, and revealing that different architectures achieve computational depth through distinct mechanisms.

The authors draw the conclusion at the level of scaling. Whether depth can serve as a meaningful axis to scale along depends, on this evidence, largely on how residual connections are designed. Get that design right and added layers become added computation. Otherwise, a deeper model at the same size may simply be a narrower one.