BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

BeaconKV. Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference. By Janghyeon Kim and three others, from Hanyang University and Sungkyunkwan University. Published on the fourth of September. Its code repository is public and real: about twenty source files.

Large Reasoning Models solve problems through long Chain-of-Thought generation, and the key-value cache that generation needs grows linearly with sequence length. For long reasoning traces it becomes a severe memory bottleneck, often exceeding GPU capacity. Existing methods compress the KV cache by using recent queries to estimate which tokens will matter in the future. That implicitly assumes recent queries are reliable proxies for future attention. The authors show the assumption fails over long horizons. Certain decoding steps produce what they call Thought Revisiting Tokens, or TRT, which re-attend to distant earlier context, such as a task-solving plan laid out early in the trace. A cache pruned by recent attention may already have discarded what those tokens reach back for.

Systematic analysis surfaced a regularity. The queries corresponding to TRT cluster into a small number of similarity groups in the embedding space. BeaconKV builds on that. It is a training-free compression method that maintains beacon queries, compact representatives for each global query cluster, and uses them to anticipate which KV pairs will be revisited.

The benefit of a small number of clusters is that a handful of representatives can stand in for the whole history. BeaconKV does not need to store every past query to know what revisiting looks like. The beacons summarise it, and they indicate which parts of the cache to keep for when the reasoning turns back. That is also why it is training-free: the beacons are maintained during inference, and the model itself is left unchanged.

The evaluation covers four open-source reasoning models and diverse reasoning benchmarks. BeaconKV generally outperforms existing compression methods. It reaches up to five point eight times memory reduction while nearly preserving the accuracy of the full cache, and it improves throughput by over four point three times.

The method rests on one observation about how long reasoning behaves. Recency is not a safe guide, because reasoning returns to its own early plans, but those returns are not scattered at random: they fall into a few recognisable groups. Keeping one beacon per group is enough to see them coming.