When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

When the KV Cache Is Slower Than the GPU. Scaling Remote KV Tiers for Context-Parallel Prefill. By Muhammad Ibrahim, from Multiverse Computing. Published on the fifteenth of October. No code repository is linked to it.

Reusing the attention KV cache across requests is a standard way to cut prefill cost in long-context LLM serving, and production systems increasingly back each engine's cache with a remote tier shared by every engine. The claim here is that such a tier can be slower than simply recomputing, and that a two-term cost model says when. A hit pays off only if the tier delivers more than a break-even bandwidth. On top of that, a fixed engine cost per hit means that short prompts cannot beat recompute at any bandwidth at all.

The study runs GLM-5.3, an FP8 mixture-of-experts model with multi-head latent attention, in a disaggregated deployment with prefill context parallelism, PCP8. Enabling the remote tier, LMCache with a Valkey store, left median time-to-first-token unchanged. But for sixteen-thousand-token requests, the p99 rose from three point nine six seconds to seventeen point three seconds.

Replaying a production request stream on an isolated engine shows why. A remote hit took zero point eight eight seconds, against zero point three one seconds to recompute the prompt. And synchronous loading turned a hit-heavy stream into an unstable queue, with a p99 of eighty-three point three two seconds while the GPUs were only seven percent busy. The cost model explains both. For this model and GPU, the break-even bandwidth is two point five three gigabytes per second. The fixed per-hit cost is about zero point three three seconds, which puts prompts shorter than roughly seventeen thousand tokens out of reach.

Guided by the model, the author makes four changes. The MLA KV store is sharded across PCP ranks, with a proof that the existing minimum-over-ranks lookup returns the longest cached prefix, and a load protocol that can never desynchronize collectives. The remote tier is partitioned by rank, one server instance per rank with no coordination, raising GET bandwidth six point three times, to ten point seven six gigabytes per second. Asynchronous loading is added to the sharded store. And three client bugs that caused unbounded stalls are fixed.

On the hit-heavy stream, with ninety-eight point five percent of prompt tokens served from the remote tier, p99 falls from eighty-three point three two seconds to zero point six seven seconds. The sharded store raises prefill throughput by fourteen percent, and by thirty-four percent together with an upstream kernel fix.

All of this is measured on one deployment of one model, and much of it on a single production stream replayed on an isolated engine. The break-even bandwidth and the per-hit cost hold for this model and GPU, so the thresholds they imply belong to this setup. Notice too where the damage showed: the median was untouched, and the cost appeared in the tail.

All changes are submitted upstream to LMCache. In one line, a cache is a saving only when fetching from it beats recomputing, and with a fixed cost per hit, short prompts lose that race no matter how fast the tier becomes.