Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

Real Long-Term Memory for AI. A 50-Million-Token Window That Is Faster and Cheaper Than Recompute. By Sietse Schelpe, from Corbenic AI. Published on the seventh of October.

A large language model can only use the text that fits in its context window, and every time a prompt is sent, the model recomputes its internal key-value state for it from scratch. The claim tested here is that this recompute can be replaced by storage. The memory layer under test is a public package called galahad-kv. It saves the key-value state of each block of about sixteen thousand tokens to encrypted local NVMe disk, and loads it back later, byte-exact, without recomputing it. The paper's title states the claim plainly: that this is faster and cheaper than recompute.

The test ran fifty million tokens of real public text through vLLM on a single NVIDIA H100, with two models: Gemma four at twelve billion parameters and Gemma four at thirty-one billion. Every block that was probed came back from the encrypted store with no recompute, one hundred out of one hundred, at depths ranging from the start of the stream to fifty million tokens in, and that held on both models.

Loading a block was between two point eight and four point three times faster than recomputing it, and used between eight point eight and twelve point three times less GPU energy. GPU memory stayed flat over the whole stream.

Then came recall. Asked about facts planted millions of tokens earlier, the smaller model answered correctly eighty-two times out of one hundred, and the larger model ninety-eight times out of one hundred. Neither model made up an answer.

The author is explicit about the limits. This is reuse of stored state, not a wider attention window. Only one block is loaded at a time, so the fifty million tokens are not all attended to at once, and how well a question is answered depends on the model, as the gap between the two recall results shows. Writing the memory is a one-time cost. And the store is not small: it takes terabytes of local NVMe disk.

The protocol is described in full, and it is built to resist common ways of gaming long-context benchmarks. A single-GPU reproduction is provided, using public software and a free licence for the package.

What the result trades is compute for disk. Instead of paying again to rebuild the key-value state of text the model has already processed, the state is written once, kept encrypted on local storage, and read back exactly when it is needed. The measured payoff is speed, energy and flat GPU memory, and the price is terabytes of disk and a model that sees one stored block at a time.