Scaling Laws for Looped Mixture of Experts

Scaling Laws for Looped Mixture of Experts. By Yanbei Chen and two others, from Meta AI. Published on the thirtieth of September. No code repository is linked to it.

There are two efficient routes to scaling a model. In a looped transformer, recurrence buys more computational depth without adding parameters. A Mixture-of-Experts model, MoE for short, uses sparsity, so it holds more total capacity at the same active compute. Existing scaling laws treat recurrence and sparsity separately. The claim here is that the two are complementary, and that their interaction can be captured in a single law. The authors call it Loop Scaling Laws, and describe it as the first to model recurrence and sparsity jointly, alongside model size and data.

At the heart of the law is a mapping from recurrence to an effective-parameter gain. It is bounded, and it is conditional on sparsity: it describes how much looping is worth in parameter terms, and how sparsity raises that gain. Fitted this way, the laws predict the held-out loss of looped models more accurately than prior alternatives. They also recover the standard dense and MoE scaling laws as special cases.

Downstream evaluations show the two axes paying off in different currencies. Sparsity delivers roughly three times the active-parameter efficiency. Recurrence yields roughly twice the total-parameter efficiency on reasoning. Scaling both together pushes the performance frontier further. The gains also hold at trillion-token scale: at matched training compute, a looped MoE whose recurrence was chosen using the law matches a non-looped MoE about twice its size on the reasoning benchmarks, and the looping enables test-time scaling through recurrence as well.

The twofold total-parameter gain is reported on reasoning, and the trillion token comparison is likewise against a larger non-looped MoE on the reasoning benchmarks, rather than across every kind of task. The law's predictive claim is about held-out loss, measured against prior alternative laws. Its practical use, as the authors put it, is as a principled foundation for designing looped MoE models under compute and memory constraints.

Recurrence adds depth while the parameter count stays put. Sparsity adds capacity while the active compute stays put. That is the sense in which the authors call the two routes complementary. What this work adds is a single equation that says how much of each to spend, and evidence that following it at scale buys a model that performs like one twice its size.