Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

Why Gated DeltaNet Survives 4-Bit Quantization. NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM. By Sergii Kozyrev and one other. Published on the third of September. No code repository is linked to it, though it does publish its weights.

Hybrid language models pair softmax attention with linear-attention layers such as Gated DeltaNet, whose recurrent state summarizes context in a fixed size. When the community quantized Qwen3.8-27B to four bits, they left the Gated DeltaNet block in higher precision, especially its decay and write-strength gates. The intuition was reasonable: errors inside a recurrence should accumulate over a long context, so the recurrent part must be the fragile part. This work tests that intuition and finds it backwards. The recurrent half is the easy half to quantize.

The test is a model called Minima, which applies NVFP4 four-bit weights and four-bit activations to all four hundred and ninety-six linear layers, Gated DeltaNet included. The model it is built on carries forty-eight recurrent layers against sixteen attention layers, so the recurrent part is not a small corner of it. The evaluation is deliberately broad, because a quantization recipe that only holds up on short inputs proves nothing about a recurrence. It covers perplexity at four thousand tokens, perplexity again at thirty-two thousand tokens, then MMLU-Pro, GSM8K, AIME, GPQA-Diamond, LiveCodeBench, and RULER retrieval out to sixty-four thousand. Across those, Minima matches full precision within seed noise, at a five-task average difference of zero point five two. It is also the smallest recipe compared, at seventeen point five gibibytes, and the fastest at prefill, by fourteen to nineteen percent. Then the detail that should not happen if the original intuition were right: the perplexity gap at thirty-two thousand tokens shrinks as position grows, rather than widening. Error that accumulates does the opposite of that. A four-part mechanism study accounts for it. First, the sixteen-element block scaling localizes the residual stream's extreme outliers, which equalizes activation error across layer roles instead of letting one role absorb it. Second, the supposedly fragile gate projections turn out to be the least sensitive part of the whole model: their softplus, exponential and sigmoid parameterizations compress around eleven percent error in the matrix multiply down to roughly two percent error at the output. Third, the delta-rule recurrence holds injected noise at a flat plateau across thirty-two thousand tokens, and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction rather than adding to what was there. Fourth, the per-token quantization cost washes out with context instead of compounding.

Two practical repairs come with it. A global-scale mismatch is fixed, the kind that appears when checkpoints calibrated per module are served by kernels that fuse those modules into a single matrix multiply. And calibrated key-value cache scales are shown to be performance-free.

The recipe reduces to quantize everything and ship the cache scales. The reason it works is the more useful part: a recurrence that overwrites its state does not accumulate error the way a recurrence that adds to it would.

Why Gated DeltaNet Survives 4-Bit Quantization. NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM. By Sergii Kozyrev and one other. Published on the third of September. This is the long version: the whole paper rather than the abstract.

This paper is about a piece of received wisdom that turned out to be exactly backwards, and it is unusually satisfying because the authors do not stop at showing it. They explain why the wrong belief was so plausible, and then they bound how far the correction travels.

Frontier open models are increasingly hybrid. Most layers replace softmax attention with a linear-attention operator whose state is a fixed-size matrix updated recurrently. In Qwen3.8-27B, forty-eight of its sixty-four layers are Gated DeltaNet and only sixteen are full attention, so the recurrent part is most of the model. Meanwhile four-bit weights and activations have become a practical serving target. NVFP4 stores each value in four bits with one shared scale per sixteen-element block, and runs natively on current tensor cores. When those two trends met, every public four-bit build of this model, including the model authors' own release, quantized the feed-forward layers aggressively while protecting the Gated DeltaNet block. In particular they protected its two gate projections: the one controlling how much of the state decays each step, and the one controlling how strongly the state is written. The argument is natural. A recurrence carries its state across tens of thousands of tokens, so per-step error should compound, and error in the gates should compound fastest.

With that suspicion in hand, the authors built Minima, which quantizes all four hundred and ninety-six linear layers of the model to four-bit weights and activations, gate projections included, leaving out only the embeddings, the head, the convolutions and the norms. Then they went looking for the reason it works, and the mechanism study is the real contribution. It runs as a chain of four findings. First, the inputs are not the reason. Gated DeltaNet reads the same residual stream as attention and sees the same extreme outliers, with hot channels a hundred times the median. But the actual per-token quantization error comes out uniform across every layer role, because block scaling confines each outlier to its own fifteen neighbours instead of letting it set the scale for a whole tensor. Second, and this is the inversion, they replayed each layer with exactly one projection quantized at a time. The two gate projections that everyone protects moved the layer output least of all, by two point one and two point six percent, even though their own matrix multiplications carried around eleven and eight point five percent error. The gate nonlinearities are a shield: the parameterization squashes a large pre-activation error into a much smaller error on the actual decay and write strengths. The error that Minima really carries comes from the three plain projections nobody was worried about. Third, injected noise into the recurrence holds at a flat plateau over thirty-two thousand tokens and a state impulse is forgotten within hundreds of steps. Fourth, the per-token cost washes out with context rather than compounding.

On the five-task accuracy average the four models span half a point, which is less than one AIME problem, and no pair is separated by confidence intervals. Minima matches the full-precision AIME score exactly, twenty-six out of thirty on all four seeds, with the entire recurrent half at four bits, and it does not compensate by generating longer: mean generation length is fourteen thousand five hundred and thirty-one tokens against fourteen thousand five hundred and thirty-two. What it buys is real. Weights drop from fifty point one three gibibytes to seventeen point five three. Time to first token on a thirty-two thousand token prefill falls from six point nine seconds to four point zero three.

Two serving-stack findings sit alongside the main result and are arguably more immediately useful. A global-scale mismatch arises when checkpoints calibrated per module are served by kernels that fuse those modules into one matrix multiply. It silently corrupts the gates, and worse, it fakes better long-context perplexity, so it flatters exactly the measurement you would use to catch it. And calibrated key-value cache scales turn out to be free at serving time while recovering eighty-three percent of the quantized model's long-context penalty.

The authors are careful about scope. One model family, one size, one quantization format, evaluated to thirty-two thousand token perplexity and sixty-four thousand token retrieval. Longer contexts are extrapolation, even though the mechanism predicts them. A concurrently released checkpoint trained under quantization rather than rounded after the fact is acknowledged but not benchmarked, because it is a different recipe class and its numbers came from a different harness. And the sharpest limit is the one on the mechanism itself: the gate-shielding argument depends on the gates being parameterized in log space through softplus and exponential. A recurrent mixer with linearly parameterized decay may get no such protection.

That last limit is what makes this a good paper rather than a good result. The finding is not that recurrences quantize well. It is that this recurrence quantizes well, for two specific reasons: gates that squash error because of how they are parameterized, and a delta rule that erases state error because it overwrites rather than accumulates. Both are checkable in any other architecture before assuming the result carries over.