HyQuant: Hybrid-Precision Quantization for LLM Attention

HyQuant. Hybrid-Precision Quantization for LLM Attention. By Jiatong Ding and eleven others, from Shanghai Jiao Tong University, Xi'an Jiaotong University, and two other institutions. Published on the twenty-eighth of August. Its code repository is worked out: about sixty-five source files, with a readme and a licence.

Quantization is now widely used in both training and inference of large language models, to reduce cost and improve efficiency. The attention module is where it gets hard. At very low bit-widths, quantizing attention often introduces large errors, and performance degrades. Existing methods mainly handle this with smoothing techniques aimed at outliers. The authors take a different route, one aimed at balancing accuracy against efficiency rather than smoothing the values that cause trouble.

HyQuant is a hybrid quantization framework for LLM attention. Its principle is that not every part of attention needs the same precision. Most attention states are quantized into low-bit formats. A small set is kept in high precision: vertical-line tokens, and the states inside a local window. These are treated as the accuracy-critical regions, and everything else can afford to be compressed.

The critical regions are chosen with lightweight signals drawn from the attention pattern, signals that are aware of vertical lines. That choice is what lets the method reduce quantization error while keeping the overhead limited. The principle is then applied to both stages of generation. During prefill, HyQuant uses a hybrid-precision quantized attention operator: vertical-line tokens and a local sliding window stay in full precision while the remaining context is quantized.

During decode, the same principle is applied to compressing the KV cache. Here HyQuant also fuses KV dequantization with the attention computation itself, rather than performing them as separate steps, and that fusion is aimed at improving both memory and hardware efficiency.

The evaluation spans diverse tasks, models, and datasets. Across them, HyQuant maintains nearly lossless accuracy. The authors stress that it does this with an extremely simple design, and they read the result as showing that hybrid quantization of attention is both efficient and practically feasible. The code is publicly available.

The whole method rests on one decision about where precision matters. Instead of reshaping every value so that low bits fit them better, HyQuant protects a small, cheaply identified set of vertical-line tokens and nearby states at full precision and lets the rest go to low bits. That same split carries from prefill into the compressed cache used at decode.