You don't need long context

You don't need long context. By Peter Belcak and eleven others. Published on the third of November.

The claim is in the title: you don't need long context. It starts from an observation about cost. Later tokens are costlier to generate than earlier ones, and on frontier open weight models the gap can reach as much as seven times. If short contexts work about as well as long ones, much of that cost need not be paid.

The tool is frequent compaction. Across a broad range of long-horizon and long-input tasks, frontier language models that maintained short contexts this way sacrificed little to no task accuracy. A sharper test came from long-horizon software engineering. Even when the model took at most one action per context, its performance did not collapse.

The scope is the one the abstract draws: frontier language models, on the long-horizon and long-input tasks tested, with the accuracy cost described as little to none rather than zero. The one-action-per-context result concerns software engineering tasks specifically, and the claim there is that performance does not collapse, not that it stays unchanged.

From that last result the authors outline a concept they call semantic state space models, or SSSMs. Their findings, as they put it, question the need for long-context language models at all.