SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan and 1 others
Tsinghua University · ByteDance Seed · M-A-P · TokenWave.AI
Published September 1, 2026 · on the feed September 2, 2026 · ▲ 90 · arXiv 2609.01343
The abstract
Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0\% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
SMELT. Scaling Laws for Compute-Matched MoE Looped Transformers. By Shaowen Wang and eight others, from Tsinghua University, ByteDance Seed, and two other institutions. Published on the first of September. No code repository is linked to it.
Looped Transformers gain effective depth by running a shared block of layers more than once. Most evaluations compare them at a fixed model size, which mixes up any architectural advantage with the extra FLOPs the looping spends. The claim here is that looping helps even when that confound is removed. The study loops Mixture-of-Experts Transformers while closely matching three budgets against an unlooped Baseline: FLOPs per token, total non-embedding parameters, and KV cache. If the looped model still wins with all three held level, the win cannot be put down to spending more.
A series of ablations leads to a recipe called SMELT, short for Sparse MoE Transformer, middle layers Loop Twice. It loops the middle half of the layers twice and still matches the Baseline on all three budgets. The authors scale it across four sizes, up to fifty-four billion non-embedding parameters, and fit a separate Chinchilla-style scaling law for each architecture.
SMELT's loss falls faster as compute grows. On the compute-optimal frontier that saves between six point eight and eighteen percent of training FLOPs. The advantage carries over to downstream benchmarks, by more than validation loss would predict. It is largest on Code, and it grows with sample length and with the number of examples given in context. Longer inputs and more in-context examples both widen the gap, rather than closing it.
The match is close rather than exact, and the result concerns this recipe, on Mixture-of-Experts models, at these four scales. Each architecture gets its own fitted scaling law, so the comparison is between curves, not single training runs. On why it works, the authors are careful. Mechanistic analysis shows the second pass through the middle layers reduces the attention sink and shifts attention mass toward tokens relevant to the content. They describe that as an inductive bias that may underlie the gains, not one shown to cause them.
The value of the study lies in its accounting. Once size, compute and cache are all held level, a looped model still comes out ahead, which means depth reuse is buying something beyond extra arithmetic. The practical output is a recipe that turns that reuse into gains you can measure.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Looping layers looks like free depth. The question is whether anything here is actually free.
You are testing a model that runs a shared block of layers more than once. Reports say such models beat plain ones of the same stored size.
The road ahead
- 1 · The problem ~6 min A model that reuses its layers looks strong, but it also spends more compute and memory, so its wins may come from spending rather than design.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Match the looped model to the Baseline on compute, parameters and cache at once, so any win belongs to the looping design.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Search loop designs under equal budgets, fix the recipe, scale it, fit a law per architecture, then compare compute at equal loss.
- 6 · The experiment ~6 min You’ll guess: A looped Mixture-of-Experts model was compared with an unlooped one at matched budgets on held-out data split by domain. In which domain was its compute saving largest?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.