Dory1.0-4B: A Looped Normalized-GPT

Dory1.0-4B. A Looped Normalized-GPT. By Maxim Fishman and eight others. Published on the second of November.

Different uses of a language model come with different inference compute budgets. The aim here is a single checkpoint that can serve all of them, spending more computation where it is needed without becoming a different model.

Dory1.0-4B is a looped, or latent-recurrent, language model with what the authors call elastic depth. The four B in its name is its parameter count, in billions. Part of the network is a recurrent block that can be applied a variable number of times at inference, so the same weights can run shallow or deep. More loops means more computation, and that computation happens in latent space rather than in longer outputs. Extra effort, in this design, does not mean writing more. It means passing the internal state through the same block again. The rest of the design follows from that choice: how the network is split up, how training teaches it when to loop deeper, and how memory is kept from growing with each extra pass.

The backbone is a normalized Transformer, nGPT, with sliding-window attention layers interleaved with full-attention layers. These are organized into an input block, a shared recurrent block, and an output block. Pre-training ran on six trillion tokens. The first four point five trillion were trained in NVFP4, and further pre-training then extended the context length to two hundred and fifty-six thousand tokens.

Post-training used supervised fine-tuning and then reinforcement learning with verifiable rewards, under a difficulty-aware loop curriculum that routes harder prompts to deeper loops. One more design choice addresses memory. During looping, Dory1.0-4B caches keys and values only from the last loop. That keeps its KV-cache footprint equal to that of a non-looped model at any loop count, while retaining most of the accuracy gains.

Increasing the loop count improves accuracy, and the largest gains land on coding, agentic, and long-context benchmarks. With two or more loops, Dory1.0-4B outperforms larger models on average across twelve benchmarks. The abstract names Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B among them. Each of those is larger than Dory1.0-4B, and the comparison is an average, not a claim of winning on every individual benchmark.

Looping also makes the model more token-efficient on agentic benchmarks. With four loops, it generates between one point three and two point five times fewer tokens than with one loop, because the extra computation is spent in latent space rather than on longer outputs. The authors release the weights in BF16, MXFP8, NVFP4, and GGUF quantized checkpoints.

Depth becomes a dial instead of a fixed property. One checkpoint, run with more loops, thinks harder internally, beats larger models on average, and writes fewer tokens doing it, while the cache cost stays where a plain model of the same size would put it. That is the bargain Dory1.0-4B offers: spend compute by repeating layers, not by lengthening answers.

Dory1.0-4B. A Looped Normalized-GPT. By Maxim Fishman and eight others. Published on the second of November. This is the long version: the whole paper rather than the abstract.

Scaling test-time compute has become a major driver of what language models can do, and almost all of that compute is spent in token space. A model thinks harder by writing more, and latency, key-value cache memory and serving cost all grow with every token it writes. Looped language models offer a second axis. Run a shared block of layers several times over the same token, and the model spends more computation per token, in latent space, without adding a single parameter. Dory carries that idea into a fully post-trained four-billion-parameter model for reasoning and agentic work. The promise is elastic depth: one checkpoint, served at whatever loop count a request can afford, with no retraining and no model swap.

Two pieces of background come first. The backbone is a normalized Transformer, called nGPT. Its hidden states are kept at unit length. Each sublayer's output is normalized, blended with the normalized residual stream using learned per-channel weights, and the blend is normalized again. Dory goes further than the original design, also normalizing the gated attention outputs and the activations inside each feed-forward layer.

The second piece is the layout. Seventy-two physical layers are split into a sixteen-layer input block, a forty-layer recurrent block, and a sixteen-layer output block. Inside every block, sliding-window attention over one thousand and twenty-four tokens alternates with full attention in a three to one ratio. The input and output blocks run once per token. The recurrent block runs as many times as the loop count says, with every weight and every learned scale shared across passes.

The recurrence is simple to state. The input block's output is held fixed. The state starts as a copy of it. Before each pass, that fixed input is added back onto the current state and the sum is normalized. The recurrent block transforms it, and the result is normalized again to become the new state, with no adapter parameters. After the last pass, the state goes to the output block and the language-model head.

Training introduces the loops late. The first four and a half trillion tokens train an ordinary, non-looped model in the four-bit NVFP4 format, which the authors report runs about twenty percent faster end to end than BF16. Looping starts only in a BF16 continuation of about one and a half trillion tokens, with the loop count drawn evenly from one to four and the same draw shared across every replica, except a rotating nineteen of one hundred and ninety-two that are forced to run exactly one loop. Gradients are truncated: earlier passes run without gradient tracking, only the final pass and the input and output blocks learn, and the loss is applied only after the final pass, with no early-exit losses.

Post-training keeps that random loop sampling through nine epochs of supervised fine-tuning, then adds the paper's most distinctive piece during two hundred steps of reinforcement learning with verifiable rewards across twenty-three environments. Rollout workers each run at a fixed loop count, and prompt quotas favour one loop, four to one to one to one. Before training, every prompt is profiled with eight responses at one loop, and only prompts neither always failed nor always solved are kept, each with its pass rate recorded. Within a batch, each prompt's rank by pass rate is mixed with a random number, with a weight of zero point three on difficulty and the rest on chance. The lowest scores fill the deep-loop slots first. Quotas hold exactly, but hard prompts are more likely to be run deep.

On an equal-weight average of twelve benchmarks, Dory scores thirty-five point two at one loop, thirty-nine point seven at two, and forty point six at four. From two loops up, that average is above Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B, and at four loops it is one point seven below Qwen3.6-35B-A3B. The gains concentrate on agentic work: SWE-Bench Verified climbs from fifty point three percent to sixty-one point eight, Terminal-Bench from twelve point five to twenty-two point two.

The curriculum has its own ablation, with and without difficulty routing. It improved all six benchmarks at both one and four loops, most at four: on Terminal-Bench, from fourteen percent to twenty-one point four.

The finding that reframes the cost is in the token accounting. At four loops, agentic tasks take one point three to two point five times fewer output tokens. On GDPval, turns per attempt fall from one hundred and sixteen to forty-nine. Counting compute as output tokens times layer applications, two loops cost only one point zero three to one point two five times a single loop, yet recover most of the four-loop gain: eighty-six percent on GDPval, eighty percent on SWE-Bench Verified. The authors conclude that two loops make a good default.

There is no limitations section, so the limits here come from the paper's own numbers. Depth does not close every gap. Qwen3.5-4B, the same total size, still beats Dory at four loops on GPQA Diamond, seventy-seven point seven against seventy-two point two, and on Humanity's Last Exam. Depth also stops paying: the base model saturates after four loops, and six or eight add nothing, so post-training stops at four.

The memory saving has a price. Last-loop caching keeps one key-value cache per layer, overwritten each loop, so earlier passes attend to final-loop entries they never saw in training. That costs up to two point one points on the long-context benchmark AA-LCR. The curriculum ablation ran on an earlier checkpoint, not the released one. Baselines were measured under the authors' own protocol, which they note may differ from published scores, and the main table carries no error bars.

The idea to carry away is depth as a dial. Instead of thinking longer in words, Dory thinks longer in layers, and the evidence says the second loop is the cheap one: it buys most of the accuracy and pays for much of itself in shorter answers.