LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

LowRAM. Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time. By Teresa Delgado and four others, from Huawei Noah's Ark Lab and University College London. Published on the fourth of November.

Autoregressive decoding depends on the KV cache, and the cache's memory footprint grows linearly with context length. That growth is what caps batch size and throughput. Existing post-training methods attack it by exploiting redundancy across tokens, across layers, or across feature dimensions, or by lowering numerical precision. Every one of those approaches looks for something to throw away: a token that matters less, a layer that repeats another, a dimension that carries little, a bit that can be dropped.

The authors identify an axis those methods leave alone. Within each attention head, the values are largely predictable from their corresponding keys. If a value can be predicted from a key you are already storing, much of the value does not need storing at all. The axis is described as complementary, which matters: it does not compete with the redundancies other methods remove, so in principle it can sit alongside them.

That observation becomes value cache amortisation. At test time, a lightweight predictor is fitted for each head, and residuals are stored only for the tokens it reconstructs worst. This gives four times near-lossless value compression. The predictor can be initialised by meta-learning, which substantially reduces the prefill overhead, or it can be fitted from scratch where a one-off cost is acceptable, multi-query inference being one such setting.

LowRAM pairs value amortisation with cross-layer low-rank key compression. The result is asymmetric: keys and values are compressed by different means. At each decoding step, only a selected subset of key-value pairs is reconstructed, which keeps decoding overhead minimal. That detail is what lets compression and speed coexist. A cache that had to be fully rebuilt at every step would save memory and spend it back as time.

Across three model families, the pipeline stays close to lossless. At four times compression, it reaches up to five times higher peak decode throughput.

It also composes with the existing approaches. Combined with quantisation and eviction, LowRAM reaches twenty-six point nine times compression while retaining ninety-four point five percent of baseline performance on long contexts.

The lesson the authors draw is structural. Keys and values sit side by side in the same cache, but they are not the same kind of object. Treated as structurally distinct, with one largely predictable from the other, they admit compression that is efficient, aggressive, and stacks on top of what other methods already do.