Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

Fathom. Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches. By Vivek Kalyanarangan. Published on the fifteenth of September. Its code repository is public and real: about twenty-five source files.

Agentic sessions can run to a million tokens, with many sessions resident at the same time. At that scale the KV cache, and the index that ranks it, live in host memory rather than on the GPU. Sparse decoding picks the top keys at each step, and to do that it scans all of the keys to rank them. That scan, the traffic of reading every key from host memory, becomes the thing that limits how fast decoding can go. In other words, the cost is not the arithmetic of attention. It is the number of bytes that have to be read just to decide which keys to attend to.

Fathom is a key scan in which each query decides for itself how many bits of each key channel to read. A query that can tell the keys apart with a coarse look reads less. The depth of reading is set per query, which is what the title means by per-query read depth.

The key cache is stored at four bits, arranged channel-major as bit planes. That layout means the first t planes of a channel are exactly that channel's t-bit quantizer, so reading fewer planes is the same as reading a coarser version of the key. Each query spends its bit budget by reverse water-filling over the importance of its channels, weighted by their variance. More bits go where they matter most.

At one million tokens on Qwen3-8B, a decode step is one point six seven times faster in GPU time than with the one hundred and thirty-six bit scans of Double Sparsity, Loki, and SparQ at r of thirty-two. Compared at equal GPU time with SparQ's sixty-eight bit read, Fathom reads eighteen percent fewer bytes, with lower attention error on six of seven model and context settings.

On RULER-style tasks, every per-token scan matches exact top-k decoding. On real coding-agent sessions, Fathom reaches the step agreement of the most accurate one hundred and thirty-six bit scan while reading ninety-two bits.

Two practical facts frame where this applies. The store Fathom reads is the four-bit copy of the keys that a quantized serving stack already holds, so no extra copy has to be kept. And the authors state plainly that the method is not faster when the index already sits in GPU memory. Its advantage belongs to the setting it was built for: long sessions, many of them, with the index offloaded to the host, where every bit not read is traffic saved.