Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

Looping Beyond Twice. A Scalable Recipe for Looped Mixture-of-Experts. By Di He and five others, from the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Peng Cheng Laboratory, and six other institutions. Published on the first of October. Its code repository is public and real: about forty source files.

Looped Transformers add recurrent depth as a new scaling axis for large language models. By applying shared Transformer blocks repeatedly, they increase effective depth without increasing the parameter count. For large mixture-of-experts models, though, the benefits of looping are unclear once comparisons are matched on FLOPs. The gains from additional iterations diminish quickly and can even turn into degradation, so the extra compute spent on looping yields little. As a result, prior work has typically settled on two loops, which is the ceiling the title refers to.

The authors identify two main obstacles to scaling looped MoE. The first is that looping inherits the curse of depth, and amplifies it. Hidden-state variance grows with each iteration as residual updates accumulate, which destabilizes deep recurrence and causes representations to drift. The second is expert selection collapse. Routers keep selecting the same experts across loops, so each extra iteration adds computation without adding any diversity to that computation. On this diagnosis, more loops means repeating much the same work on an increasingly unstable state.

LOOM is built on a single principle drawn from that diagnosis: each loop should contribute new computation, while the recurrent state is kept stable. The two halves of the principle answer the two obstacles one for one. Stability is the answer to the curse of depth, and new computation is the answer to expert selection collapse.

For stability, LOOM scales residual updates so that variance growth is bounded, and it re-injects the input embedding at every loop, so each pass starts from a reminder of the original input. For diversity, it gives each loop its own router, so that different loops engage different experts. It also adds a Looping Residual, which carries earlier outputs forward. Read against the diagnosis, the four pieces divide cleanly. Scaled updates and the re-injected embedding keep the state from drifting as loops pile up. Per-loop routers and the Looping Residual make sure that a later loop has something new to do, rather than repeating the computation of the one before.

Experiments cover models from one hundred million to one point seven billion parameters, and show stable scaling to between nine and twelve loops. Under near-iso-FLOP conditions, the seven hundred million parameter model performs best at five loops. Against the non-looped baseline, it reduces perplexity from eighteen point three six to sixteen point five four. Its average zero-shot accuracy rises from thirty-eight point eight four percent to thirty-nine point five three percent.

Without FLOP matching, the gains are larger. The one point seven billion parameter model, trained on sixty billion tokens, peaks at nine loops. There it reduces perplexity from nine point six two to seven point seven seven, and improves average zero-shot accuracy from forty-two point four percent to forty-seven point seven percent.

The contribution is a recipe rather than a single component. Its claim is that looping scales past twice once each pass is both safe to repeat and different from the last, and those two conditions map directly onto the two failures the diagnosis found. In these experiments, meeting them moves the best number of loops from two to five under matched compute, and to nine without that constraint.