Cortex: Consolidating Context into Parametric Memory
Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang
Zhejiang University · ByteDance Seed · Tsinghua University
Published October 14, 2026 · on the feed October 15, 2026 · ▲ 67 · arXiv 2610.20399
The abstract
Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
The Brief
1:54A short podcast summary of the paper. Written and voiced by AI.
Transcript
Cortex. Consolidating Context into Parametric Memory. By Jintian Zhang and seven others, from Zhejiang University, ByteDance Seed, and one other institution. Published on the fourteenth of October. It links to a code repository that is no longer there.
Giving a large language model the right context can improve its performance, but the improvement may vanish as soon as the context is taken away. Taking inspiration from systems consolidation in biology, the authors name the task of keeping it: Memory Consolidation. The goal is to internalize the gains that context brought, while preserving the capabilities that were not the target. Framed this way, context-specified editing, skill learning, and behavioural steering all become instances of one common objective.
Cortex approaches consolidation in two moves. It generates prospective queries that anticipate how the context will be used. It then distills the outputs the model gives when conditioned on the context into updates to the model's weights, updates that are mergeable.
Those updates come in two forms: low-rank updates, or structural updates to the feed-forward network. To measure the result, the authors introduce a testbed of five task families, which measures both consolidation and retention under single, batch, and sequential updates.
Backbones range from four billion to one hundred twenty billion parameters, dense and mixture-of-experts, text-only and multimodal. In most of the settings evaluated, Cortex lies on the empirical Pareto frontier between consolidation and retention. After fifty sequential factual edits on a fourteen billion parameter model, Cortex-Structural recovers eighty-eight point nine percent of the gain the context had produced, while retaining ninety-nine point eight percent of the original performance on everything else.
The two numbers in that last result are the two halves of the objective. One says how much of what the context taught has stayed. The other says how little of what the model already knew was disturbed along the way.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A model that forgets the moment you close the document. That is the wall.
You hand a language model a document, a new tool's instructions or a persona, and it does much better with that text in its prompt.
The road ahead
- 1 · The problem ~7 min A language model does better with helpful text in its prompt, but loses that gain the moment the text is removed.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min The model reading the context acts as a teacher, and a context-free copy learns its answers on anticipated queries through small mergeable feed-forward additions.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Cortex generates anticipatory queries, answers them with the context in view, then distils those answers into mergeable feed-forward updates.
- 6 · The experiment ~6 min You’ll guess: After fifty sequential factual edits on a 14B model, what share of the context-induced gain did Cortex-Structural recover?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.