Cache Engineering for Agents

Cache Engineering for Agents. By Tianyu Fu and eight others, from Tsinghua University and the Chinese University of Hong Kong. Published on the twentieth of October. Its code repository is worked out: about fifty source files, with a readme and a licence.

Language-model agents now take on complex tasks that unfold over long trajectories of observations and actions. As a trajectory grows, the input and output traffic and the storage of the key-value cache become the dominant cost. A trajectory can also outrun the model's maximum context length, so the growth of the active cache has to be controlled somehow.

Prior work does this through context engineering, trimming or summarising the history to shrink the cache. The catch is that editing the start of a trajectory invalidates every cached state that came after it. The retained context then has to be prefilled again, which is costly. And if an agent keeps very little context to limit that re-prefill cost, task accuracy can suffer. Either way, something is paid.

Cache engineering, the scheme proposed here, lets an agent control its active key-value cache directly, without modifying the trajectory and without re-prefilling the context it keeps. Two operations make that possible. An agent can use drop to exclude selected cache from future attention, and repos to adjust the position encodings of the keys it retains. The agent applies them proactively, as part of how it works.

Formalising these operations exposes a serving problem the authors call history dependence. Identical token sequences can end up with different cache states, depending on which sequence of cache operations produced them. Prefix matching, which ordinarily decides when a cache can be reused, is no longer enough to guarantee the reuse is correct. Two requests may look the same token for token and still need different caches, because one agent dropped part of its history and the other did not, or one repositioned its keys and the other left them. The server can no longer tell from the text alone which cache state a request belongs to.

The answer is a co-designed serving engine called AKV. From the outside it presents a stateless interface compatible with existing serving APIs. On the inside it manages the stateful parts itself: cache reuse, eviction, and restoration. Clients keep calling it the way they already call a serving engine, and the bookkeeping that history dependence demands happens behind that interface.

The comparison is against conventional threshold-triggered summarization. A tool-response rolling-drop cache policy improves average benchmark scores by one point seven points while cutting average cost by twenty-seven point seven percent. Served through AKV, it reaches up to one point two times the output throughput and a one point five times speedup in P95 request latency. So the policy is cheaper and slightly more accurate than summarising, and the engine makes it faster to serve as well.

Cache policies can also improve themselves. Recursive self-improvement of the policies reduces cost by a further seven point one percent beyond an expert-designed policy.

The shift is in where the editing happens. Context engineering edits the text, and the cache has to be rebuilt to match. Cache engineering edits the cache, and the text is left alone. That move creates a new obligation for the server, tracking how each cache state came to be, and AKV is built to carry it while looking like an ordinary serving endpoint from outside.