STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

STEPQuant. When and Where Errors Matter in Delta-Rule Recurrent State Quantization. By Bingchen Yao and eight others, from Zhejiang University, the Institute of Automation, Chinese Academy of Sciences, and four other institutions. Published on the twenty-ninth of September. Its code repository is worked out: about eighty-five source files, with a readme.

Linear attention replaces the growing KV cache with a recurrent state of fixed size. Under concurrent serving, though, those persistent states add up, and can become a substantial memory bottleneck. The obvious remedy is to quantize the states directly to low precision, and that often degrades accuracy severely, because quantization errors propagate through every successive state update. The appeal of linear attention is that memory stops growing with the sequence. A fixed size is not the same as a small one, though, once many states are held at the same time.

The authors find that the impact of those errors depends on two complementary dimensions. In time, errors in long-lived memory can persist across many decoding steps. In space, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Put together, an error is not just an amount. It has a duration, how long it survives in the state, and an address, which row it sits in, and both change what it costs. STEPQuant is a spatial-temporal post-training quantization framework for Delta-rule recurrent states, built on both observations.

It allocates precision according to two quantities: the magnitude of the error, and the lifetime of the memory. It also fits scales for key rows and value columns jointly, based on the state distributions and on how strongly each key row affects output error. Precision, in other words, is not spread uniformly. Parts of the state whose errors would be large or long-lived get more of it, and the row and column scales are fitted together rather than one direction at a time, because magnitudes vary along both.

Experiments use Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct, across both long and short generation benchmarks. Under a nominal six-bit budget, STEPQuant closely matches the accuracy of FP32 states, and its four-bit configuration outperforms uniform INT8. The two bit widths tell two parts of the story: six bits is where accuracy is kept, and four bits is where it still beats a uniform eight-bit scheme. Integrated into SGLang with optimized GPU kernels, six-bit STEPQuant achieves more than five times compression of the recurrent state, and reduces total serving memory by up to sixty-eight point seven percent.

The title asks when and where errors matter, and the method is that answer turned into a budget. Bits go where an error would live a long time, or land on a key row that moves the output, and the scales follow the shape of the state along both of its axes.