Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Vishesh Tripathi, Abhay Kumar, Ramsha Khan
FrontiersMind
Published September 8, 2026 · on the feed September 15, 2026 · ▲ 81 · arXiv 2609.13285
The abstract
The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with sequence length. Grouped-query attention (GQA) reduces this cost by sharing key-value heads, but still stores both a key and a value at every step. We introduce Grouped Value Attention (GVA), which stores grouped values and reconstructs content keys with a learned linear map. At inference, the map can be absorbed into the query, eliminating the need to materialize content keys in the intended decode path. A small shared decoupled RoPE channel retains positional information through a separately cached positional key. For the configurations studied, this representation reduces persistent cache scalars by approximately 45-47% relative to matched GQA. At the 350M-parameter scale with 30B FineWeb-Edu tokens, the 16-dimensional positional variant reaches 44.18 average accuracy across five tasks, compared with 44.36 for GQA and 43.88 for MLA. These results demonstrate near-GQA benchmark accuracy with a more compact cache representation. To translate this compact representation into faster autoregressive inference, we have developed custom decoding kernels and are currently evaluating their end-to-end inference performance with an open-source release planned soon.
The Brief
2:25A short podcast summary of the paper. Written and voiced by AI.
Transcript
Grouped Value Attention. Efficient KV Caching via On-Demand Key Reconstruction. By Vishesh Tripathi and two others, from FrontiersMind. Published on the eighth of September. No code repository is linked to it.
In Transformer decoding, the KV cache is a primary bottleneck. Its memory footprint and its cache-read traffic both grow with sequence length. Grouped- query attention, or GQA, reduces that cost by sharing key-value heads, but it still stores both a key and a value at every step. Every generated token adds to that cache, so a long sequence means more memory to hold and more to read back at each decoding step. Storing less per step saves on both counts at once.
Grouped Value Attention, or GVA, stores only grouped values. The content keys are not stored at all. They are reconstructed with a learned linear map. In place of two stored quantities per step, there is one, and the other is computed from it when it is needed.
At inference the map can be absorbed into the query, which removes the need to materialize content keys in the intended decode path. Position is handled on the side. A small shared decoupled RoPE channel keeps positional information, through a positional key that is cached separately. For the configurations studied, this representation reduces persistent cache scalars by approximately forty-five to forty-seven percent relative to matched GQA. Content and position are thus split. Content keys come from values through the map, while position travels in its own small channel with its own cached key.
The comparison is made at the three hundred fifty million parameter scale, with thirty billion FineWeb-Edu tokens. The variant with a sixteen-dimensional positional key reaches forty-four point one eight average accuracy across five tasks. GQA reaches forty-four point three six, and MLA forty-three point eight eight. The authors read this as near-GQA benchmark accuracy with a more compact cache representation. All three figures sit within half a point of one another. GVA trails GQA slightly and leads MLA slightly, while caching markedly less than GQA.
One piece is not yet in hand. To turn the compact cache into faster autoregressive inference, the authors have developed custom decoding kernels, and they are still evaluating end-to-end inference performance, with an open- source release planned soon. For now the result is about cache size and accuracy. Speed is the next measurement, and the kernels are what it will be taken on.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every token you generate makes the cache bigger. Here is the wall.
You are serving a Transformer that writes long answers. Every new token adds a key and a value to the cache, and each step reads the whole cache back, so memory and read traffic grow with length.
The road ahead
- 1 · The problem ~6 min When a Transformer writes long answers, the stored keys and values from past tokens grow with every step, and reading them back each step becomes the bottleneck.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Cache only the grouped values, rebuild each head's content key from its value with a learned linear map, and keep position in a small separate channel.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Train with a learned value-to-key map, fold that map into the query for decoding, then score straight against cached grouped values.
- 6 · The experiment ~6 min You’ll guess: At the 350M-parameter scale, how did GVA's average accuracy compare with GQA and MLA?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.