Memory Tax: Dissecting the Net Value of Agent Memory

Memory Tax. Dissecting the Net Value of Agent Memory. By Xuan He and ten others, from the University of Illinois Urbana-Champaign, Tsinghua University, and three other institutions. Published on the twenty-fifth of October.

Long-horizon agents increasingly come with memory systems. An agent could instead simply retrieve directly from its raw history, and whether building a memory is worth more than that has remained largely unclear. The claim here is that, in many cases, it is not. Many dedicated memory systems underperform naive retrieval-augmented generation, plain RAG, which means their net value is negative. To explain why, the authors split net value into two parts, measured against RAG as the reference. The gains over direct retrieval they call memory benefit. The losses they call memory tax.

The starting observation comes from a large-scale evaluation spanning multiple benchmarks and multiple model backbones. Across it, many memory systems fall behind naive RAG. The account the authors give is about what a memory process does to information. It changes how the raw information is organized and how it is accessed. They describe those changes as double-edged deviations from the raw history. Each was designed to help the task, by reorganizing facts, say, or by distilling preferences. But some of the same changes can remove evidence or obscure it, and that loss is the memory tax.

Because memory benefit is simply the gain a design was built to deliver, the authors give most of their attention to the side that tends to be overlooked. They study how memory tax arises from the system side and from the evaluation side, and then generalize the concept to further application scenarios for memory. In other words, the tax is not treated as a quirk of one system. It is traced to choices in how memory systems are built, to choices in how they are evaluated, and then followed beyond the settings where it was first observed.

The negative net value is reported for many systems in many cases, not for every memory system everywhere. The yardstick is direct retrieval from raw history with naive RAG, and benefit and tax are both defined relative to it. The evidence is the evaluation across the benchmarks and backbones studied. Nor is the claim that memory is useless. Memory benefit is real, and it is what these designs set out to deliver. The point is that benefit is only half of the account, and a system can pay more in tax than it earns.

The authors hope memory tax becomes a lens: one that moves attention away from how a memory system performs on its own and toward its net value over the simplest alternative. Better memory designs, and better evaluations of them, are what they expect that shift to produce. A memory that reorganizes the past may also be hiding part of it, and the cost of that only shows when you compare against the past left alone.