WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms
Jiale Chen, Vage Egiazarian, Eldar Kurtić, Torsten Hoefler, Dan Alistarh
ISTA · Red Hat AI · ETH Zürich
Published September 29, 2026 · on the feed October 1, 2026 · ▲ 9 · arXiv 2609.38121
The abstract
KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low-bit KV-cache quantization. It adapts WUSH, which constructs a data-aware transform from the second-order statistics of both factors in a matrix product to reduce quantization error. WUSH-KV uses calibration data to construct separate key and value transforms, with the value transform folded into the model weights and the key transform applied after RoPE. The transforms can be paired with clipped quantizers. For one such quantizer, QuEST INT, we show that, under mild assumptions, the WUSH transform is near-optimal. With this quantizer, WUSH-KV reduces layerwise reconstruction error and achieves the lowest end-to-end perplexity among other tested transforms. For end-to-end evaluation, we integrate WUSH-KV into SGLang using OSCAR-style percentile-clipped affine quantization. At 2-bit, WUSH-KV performs comparably to or outperforms the OSCAR transform across all evaluated models and downstream tasks.
The Brief
2:05A short podcast summary of the paper. Written and voiced by AI.
Transcript
WUSH-KV. KV Cache Quantization with Data-Adaptive Transforms. By Jiale Chen and four others, from the Institute of Science and Technology Austria, Red Hat AI, and one other institution. Published on the twenty-ninth of September. No code repository is linked to it.
Long-context inference has a memory problem that scales in two directions. The KV cache costs memory and bandwidth, and both grow with context length and with batch size. That bottleneck limits how efficiently a model can work over long inputs. Storing the cache at very low bit widths is one way through, provided the quantization error stays small.
WUSH-KV targets that error with a transform built from the data. It adapts an existing technique, WUSH, which constructs a data-aware transform from the second-order statistics of both factors in a matrix product, in order to reduce quantization error. Here the idea is carried over to the key and value cache.
Calibration data is used to build two separate transforms, one for keys and one for values. The value transform is folded into the model weights. The key transform is applied after RoPE. Either can be paired with a clipped quantizer, and for one such quantizer, QuEST INT, the authors show that under mild assumptions the WUSH transform is near-optimal.
With that quantizer, WUSH-KV lowers layerwise reconstruction error and reaches the lowest end-to-end perplexity among the transforms tested. For an end-to-end system test, it is integrated into SGLang using OSCAR-style percentile-clipped affine quantization. At two bits, WUSH-KV performs comparably to or better than the OSCAR transform across every evaluated model and downstream task.
Here the lever is the transform in front of the quantizer, and the quantizer itself is borrowed. A transform fitted to calibration statistics, split by keys and values and placed so that one half disappears into the weights, gives a two-bit cache that holds up against the transform it was measured against.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a memory store that grows with every token you read.
You are serving a language model on long documents with large batches. Every attention layer stores the keys and values of past tokens, and that store grows with length and batch size and is read again at every new token.
The road ahead
- 1 · The problem ~7 min The key-value cache of a long-context model is huge, and squeezing it into 2 bits adds errors that travel through attention in two different ways.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Build each change of basis from data: how the cached vectors vary and how their consumer turns errors into output error, separately for keys and values.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Gather calibration statistics, build key and value transforms per head, fold the value one into weights, apply the key one after RoPE, then quantize.
- 6 · The experiment ~6 min You’ll guess: With 2-bit KV caches, how did WUSH-KV compare with the OSCAR transform across the evaluated models and downstream tasks?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.