PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

PACE. Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States. By Patrick Haller and two others, from Humboldt University of Berlin. Published on the twenty-first of October. No code repository is linked to it.

A linear-attention language model keeps its context in a recurrent state of fixed size. An inference system stores one copy of that state for every sequence it is decoding at the same time. Store the state at lower precision and more sequences can share one accelerator. Existing state quantizers round the state after every token that is generated, and they report that four bits lose accuracy unless the bit width goes up or the model is fine-tuned.

PACE is a post-hoc four-bit state quantizer, built on two observations about where the error comes from. The first concerns timing: rounding less often reduces the error that accumulates in the state. The second concerns structure: in the gated-delta state, the large entries gather in columns.

For timing, PACE rounds the state only once every few generated tokens. Chunked kernels already do something similar during prefill, rounding once per chunk. The tokens generated in between are kept in a small buffer until the next rounding. The state therefore passes through the quantizer far fewer times over a long generation, and each pass is a chance for error to enter. Fewer passes, by the authors' first observation, means less error building up in the state.

For structure, the authors trace the column pattern to the update written at each token: some of its channels remain large across many tokens. So PACE shares each quantization scale within a single column, aligning the scale with the channels. One further choice handles the hardest cases. The heads that retain rounding error the longest are kept at eight bits rather than four. Those are the places where an error, once made, would linger, so they get the extra precision while the rest of the state stays at four bits.

The tests use four hybrid models, from four billion to thirty-five billion parameters. With mixed heads, some at four bits and some at eight, PACE is competitive with eight-bit DAMP on the three larger models, while using about two thirds of its memory. With four-bit heads throughout, PACE scores higher than Q-Mamba at equal memory. And rounding only every few tokens preserves the decoding speedup that the buffer brings. Those are two different comparisons: against an eight-bit method, PACE is competitive on the larger models while using about a third less memory; against a four-bit method, it matches the memory and scores higher.

The name carries the design. Periodic, for rounding once every few tokens instead of after each one. Channel-aligned, for sharing quantization scales along the columns where the large entries collect. Together with eight bits for the few heads that hold on to error longest, those two choices are what let four bits work post hoc, without raising the bit width everywhere and without fine-tuning the model.