REMORY: Learning Residual Memory for Context Compaction

REMORY. Learning Residual Memory for Context Compaction. By Hanchen Xia and seven others, from the Shanghai Academy of AI for Science, Fudan University, and six other institutions. Published on the eighth of October.

An agent working over a long horizon eventually runs out of context window. To keep going, it compacts its history, usually into a textual summary. The trouble is that a summary chosen once may not support every decision that comes after it. Whatever the summary left out is simply gone.

REMORY keeps the summary and adds something next to it. It is a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the full history and the summary, the network learns to produce tokens that help a frozen language model approximate the continuation it would have produced if it still had the full history in front of it.

Placement is the design choice. The memory tokens are conditioned on the summary and appended after it. The authors describe this as an analogue of a residual connection, running along the sequence dimension instead of through depth. The summary carries the main signal, and the learned tokens carry a correction on top of it, while the language model itself stays frozen.

On SummHay, REMORY improves source attribution while insight coverage stays nearly unchanged, and it approaches the full-context joint score using only five point two percent of the input positions. Across long-horizon agent benchmarks, both Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also produce substantially fewer repeated tool outputs and fewer tool errors on BrowseComp and on Terminal-Bench 2.1.

Compaction usually means choosing what to lose. REMORY reframes it as a summary plus a learned residual, a small set of tokens whose whole job is to make the compacted context behave more like the full one. The fall in repeated tool calls is a concrete sign of what that buys an agent.