Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design

Towards Looped Models Done Right, Part I. Topology, Input Injection, Recurrent-State Design. By Benhao Huang and six others, from the Institute of Foundation Models, the University of Southern California, and one other institution. Published on the sixteenth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.

Looped language models reuse the same physical Transformer blocks across logical depth. Established architectures tie three design choices together: where the recurrence is placed, how the input is injected, and how the recurrent state is organised. Pulling those choices apart, the authors find that they do not pull in the same direction, and that taken together the results favour the Huginn-style design.

Each axis is isolated in models trained from scratch, matched for parameter scale, logical depth and token budget. A sandwich architecture, with a prelude, a recurrent block and a coda, consistently improves context extraction and reasoning-intensive tasks. Input injection mainly helps knowledge retrieval and context utilisation. Random initialisation, and separate H and L recurrent states, give mixed or negative returns. At seven hundred and thirty million parameters and three hundred and thirty-six billion tokens, the complete Huginn architecture beats Ouro on all ten benchmarks.

The principles were then tested on Mixture-of-Experts models, matched for both resident and active parameters. At the scale written as eight B A zero point eight B, or thirty-two B A three B when unrolled, trained on five hundred billion tokens, Huginn outperforms Ouro on nearly all benchmarks across knowledge, reasoning and coding, with more balanced expert routing too. Against a one hundred and twelve layer feedforward MoE baseline, under matched training and inference FLOPs, the Huginn-style model uses seventy-five percent fewer resident parameters. It still beats that baseline on DROP and GSM8K, matches it on MATH500, and stays competitive elsewhere.

These are design principles drawn from controlled comparisons at the scales named, and the verdicts are not uniform. Huginn beats Ouro on nearly all benchmarks at the larger scale, not on every one. Against the feedforward baseline, the result is a win on two benchmarks, a tie on one, and competitive on the rest, rather than a clean sweep. And as part one of a series, it covers topology, input injection and recurrent state, leaving other questions about looped models for later.

If the series promises looped models done right, part one's answer is fairly specific. Put the recurrence in the middle of a prelude and a coda, inject the input, skip the separate recurrent states, and the result holds up when the model becomes a mixture of experts, with far fewer parameters resident in memory.

Towards Looped Models Done Right, Part I. Topology, Input Injection, Recurrent-State Design. By Benhao Huang and six others, from the Institute of Foundation Models, the University of Southern California, and one other institution. Published on the sixteenth of October. This is the long version: the whole paper rather than the abstract.

A looped language model runs the same Transformer blocks again and again, buying depth with computation instead of with stored weights. The appeal is practical: compute throughput on modern hardware is growing much faster than memory capacity and bandwidth. Most looped models descend from one of two recipes, called Ouro and Huginn, and the two differ in several ways at once. When one beats the other, nobody can say which difference did the work. This study pulls the recipes apart, one choice at a time, with everything else held fixed.

Three choices are bundled together. The first is the iteration envelope: where the loop sits. Ouro loops the whole stack. It stores twenty-eight blocks and runs all of them four times. Huginn wraps the loop in a sandwich: eight untied prelude layers run once, then a tied core of twelve layers runs eight times, then eight untied coda layers run once. Both come to one hundred and twelve logical block executions from twenty-eight stored blocks. The second choice is input injection: whether each pass of the loop gets the input written back in. Ouro does not. Huginn writes the prelude's output into the loop state before every pass. The third is how the loop state is organized and started. Ouro starts it from the token embeddings. Huginn starts it from random noise.

Every dense model here stores seven hundred and thirty million parameters and is trained from scratch on the same TxT360 data. Two plain feedforward models anchor the comparison: one with twenty-eight blocks, matching stored size, and one with one hundred and twelve, matching logical depth.

The experiment is a construction path from Ouro to Huginn, one step at a time. Step one changes only the envelope. Step two adds the input write. The write is a simple per-channel operation: the loop state is shrunk by a learned decay, and a projected copy of the input, scaled by a learned gate, is added on top. In the sandwich, the thing written is the prelude's contextualized output. Ouro has no prelude, so its matching control writes the raw token embeddings. Either way, each write is compared against the same model with no write.

Step three varies the state. One variant swaps the input-started state for a random one, which completes the standard Huginn recipe. Another tests a two-level hierarchy borrowed from hierarchical reasoning models: a high-level state started from the input, a low-level workspace started from noise, and a schedule of two high cycles, each with three low updates. Both states pass through the same shared body, eight applications in all, so compute stays matched.

The envelope does the heavy lifting. Moving from full-stack looping to the sandwich raises MATH five hundred by twelve points, from twenty point eight to thirty-two point eight, and GSM8K by seven. Knowledge-heavy and code tasks see no consistent gain. Input injection pulls the other way: it helps tasks that lean on the prompt, with BBH chain of thought up six point six and HumanEval plus up five point five, but in the sandwich it costs three point six points on MATH five hundred. The authors suggest that repeatedly writing the input back may keep pulling the state toward where it started, when maths needs it to travel. Even so, the injected sandwich beats injected Ouro on eight of ten benchmarks.

The state experiments come out flat or negative. Random initialization gives two gains and four losses larger than a point, and the input-started state wins on six of ten. The two-level hierarchy drops MATH five hundred by seven point two points.

The design then moves to mixture of experts, with eight billion stored and about eight hundred million active parameters per pass, trained on five hundred billion tokens. Huginn beats Ouro on eight of ten benchmarks, by the widest margins on GSM8K and MATH five hundred. Its experts are used more evenly. The most direct test of what looping does with experts is an intervention: force iterations two through eight to reuse the experts chosen on the first pass, and accuracy drops on all six tasks tried, by eleven point two points on MMLU. A feedforward model of the same logical depth, with four times the stored parameters, still leads on seven of ten, but the average gap shrinks from about five points in the dense setting to under two.

There is no limitations section as such. A future work section names the gaps plainly. Everything runs at one practical scale, so nothing here is a scaling law. Each result is a single run; multi-seed replication is listed as still needed, and many of the gaps are a point or two. The learning rate was swept and the best setting reported. The mixture of experts comparison pits two complete recipes against each other, so it cannot say which ingredient made the difference there. The hierarchy test shares one module across both levels; a version with separate modules is still under evaluation and left out. The random-state test does not measure path independence or behaviour beyond the trained loop count, which is where random starts are usually argued to help. And the injection trade-off is shown, not explained: its mechanism was not isolated.

The idea to carry away is that a loop is not one design decision but several, and they pull in different directions. Where the loop sits mattered most here, and helped reasoning. Writing the input back helped tasks that need the prompt and cost a little on maths. Extra state structure added complexity without a reliable return. Compare looped models only after matching storage, depth, data and evaluation, and attribute their differences one choice at a time.