History Replays Itself: Unleashing Long-Context Capability in Linear Attention

History Replays Itself. Unleashing Long-Context Capability in Linear Attention. By Li Nanbo and twelve others, from King Abdullah University of Science and Technology. Published on the eleventh of October. No code repository is linked to it.

Linear attention promises the efficiency that long-context language modelling needs, but its recurrent state has a fixed size, and that one state must do two jobs at once. It has to support the computation in progress, and it has to preserve information from a history that keeps growing. That coupling has reinforced a prevailing view: the memory capacity of the state defines the model's long-context capability. The authors challenge that presumed ceiling. Recurrent computation, they argue, need not bear the full burden of long-term storage.

NoteBank takes storage off the recurrent state's shoulders. It is a sparse episodic memory that archives selected text before any future query is known. It uses the backbone's own native features both for writing to that archive and for addressing it. And when evidence is retrieved, the model reads it through its native forward computation, by replaying it.

Because every step leans on what the backbone already has, the interface is training-free and plug-and-play. It needs no external embedding model, no separately trained retriever, and no separate neural reader. Writing, addressing and reading are all done by the pretrained model itself, and only selected text is stored, so the storage stays sparse. The division of labour is the point. The recurrent state goes on doing the computation it was built for, while the archive holds the history the state could never fit. When a question finally arrives, the relevant evidence is not reconstructed from a compressed summary. It is replayed, and the model reads it again the same way it read it the first time.

At one million tokens, NoteBank raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from one point seven percent to seventy point three percent. No significant retrieval degradation is observed anywhere from four thousand tokens up to one million.

The cost is small. The recovery adds at most zero point zero two five percent to analytical FLOPs, and persistent context storage is roughly two thousand six hundred times smaller than a Transformer's full multi-head attention KV cache, at matched width, depth and precision. Experiments extend from three recurrent backbones of one point three billion parameters to Mamba-2-8B and a post-trained Qwen3.5-9B, with gains on both versions of LongBench and on MemoryAgentBench.

The result moves the question. If evidence is stored sparsely and replayed through the model's own forward pass, the size of the recurrent state is no longer the whole story for long context, and the efficiency of recurrent computation is kept. Memory capacity and long-context capability, treated as one thing, come apart.