Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

Working Around the Compute Ceiling. Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost. By Sietse Schelpe, from Corbenic AI. Published on the thirtieth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

A transformer language model does a bounded amount of computation per token. Recent work by Vishal Sikka, formerly chief executive of Infosys, argues that this bound limits which tasks a model can carry out or verify. This paper takes the ceiling as given and asks a narrower question: how much of the budget beneath it goes on work the model has already done? Serving is stateless from one request to the next. Ask a second question about a document, and the model recomputes the document's attention state starting again from the first token. Across seven real-world datasets, ninety-eight point seven percent of prompt tokens were text the model had already read. Put the other way round, only a sliver of each prompt was new. Almost all of the computation spent on reading was spent on reading again, and that is the share of the budget this work goes after.

Galahad is a memory layer for vLLM, SGLang and llama.cpp whose purpose is to make that reading a one-time cost. It has two parts. Taliesin saves the model's key-value state for a block of text, and when a later request contains the same bytes, it loads that state instead of recomputing it. Blaise keeps the documents themselves, and hands the model only the section that a given question needs. The two divide the work. One avoids reading the same bytes twice, and the other avoids reading what the question does not require.

Exactness is the design constraint. Restored state is bit-identical to what recomputation would produce: after a restart, after rehydration, and after a hot-load, all two hundred and sixty-two thousand one hundred and forty-four output logits matched. Galahad also fails closed. Any load that does not pass its checks is simply recomputed, so a bad cache never reaches the model. Under vLLM it worked with all thirty models the author tested.

The main test is a recall task: one hundred facts hidden in a corpus of ninety-seven thousand tokens, using Gemma 4 at thirty-one billion parameters. On llama.cpp, Taliesin alone let the model attend to the whole corpus. It answered ninety-eight of the hundred, at three seconds and five hundred and seventy-two joules per question. The same model without Galahad could hold only the last twelve thousand tokens. It answered ten, taking nine point three seconds and two thousand seven hundred and fifty-four joules per question.

Adding Blaise changed the picture again. The model read about six hundred and sixty-eight tokens per question and answered every one of the hundred, on all three runtimes, in roughly six tenths of a second and around two hundred joules each. A tuned RAGFlow pipeline, for comparison, answered seventy-seven.

The up-front cost is storing the corpus once: about one hundred seconds and twenty-eight kilojoules. That energy is earned back after thirteen questions. Every question after that is cheaper than it would have been without the stored state, both in time and in energy, and the corpus never has to be read from scratch again.

The claim is less about beating the compute ceiling than about not wasting what lies under it. If almost every prompt token has been read before, then reading should be paid for once and remembered exactly, with any doubtful memory thrown away and recomputed. Taken together, the author argues, these results move language model serving from stateless inference to stateful inference.