Context Language Models

Context Language Models. By Rulin Shao and twelve others, from the University of Washington, Meta Superintelligence Labs, and two other institutions. Published on the twenty-ninth of September. Its code repository is worked out: about forty-five source files, with a readme and a licence.

A language model working through a long task has to decide what stays in its context and what goes. Usually that decision is not the model's to make. Context management sits in an external harness, a set of strategies wrapped around the model that control what it gets to see. The model itself never learns what is most important to keep.

Context Language Models, or CLMs, are language models that natively manage their own context. The implementation is simple to state. The context is treated as a file, and the model is allowed to make unrestricted updates to that file. Because the model is the one editing, it can learn what is most important to maintain. The same framing extends naturally to multi-agent systems, where several agents' contexts coexist side by side as files.

Moving context management from external harness control into intrinsic model behaviour opens two routes for learning it. One is in context: CLMs can be steered with natural-language instructions, evolved through a standard skill-optimisation loop. The other is parametric: the authors introduce an online reinforcement learning method for CLMs. Serving is co-designed too, with a technique called Suffix Cache Reuse aimed at the server-side cost of running models that rewrite their own context.

Even built zero-shot from existing models, CLMs outperform state-of-the-art context management strategies across a variety of tasks. On BrowseComp-Plus, accuracy is eleven point four percent higher with twenty-one point five percent fewer FLOPs. On a twelve-hour EdgeBench run, scores are five percent higher with fifty-nine percent fewer FLOPs. And on a twenty-four-hour agent-swarm task spanning multiple repositories, the improvement is sixty-five percent greater at the same compute.

Learning the strategy adds more. The evolved instructions improve held-out accuracy on a context-management task by up to thirty-five point nine points, while reducing compute. The reinforcement learning method lifts Qwen3.5-9B on BrowseComp-Plus by forty-seven point six percent, using twelve percent fewer FLOPs. On the serving side, Suffix Cache Reuse cuts server-side compute by thirty-five percent relative to standard SGLang, at matched performance.

Most of the gains here come in pairs: better results alongside fewer FLOPs. Underneath them is a change in where the decision lives. Once managing context belongs to the model rather than to the harness around it, it becomes something that can be taught, by instruction or by reward, and served more cheaply when the serving stack is built with it in mind.