The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

The Evolution of Attention in Large Language Models. Mechanisms, Trade-offs, and Emerging Trends. By Zhentao Tan and five others, from Alibaba Group. Published on the thirtieth of September. No code repository is linked to it.

Self-attention gives a large language model fine-grained access to its context, and that access depends on the query. It also has costs. Dense interaction between tokens makes the prefill cost quadratic, and the key-value cache grows as the context gets longer. Research in response has spread in several directions: compressing explicit memory, accessing it sparsely, constructing recurrent state, modelling structured state dynamics, and composing heterogeneous mechanisms together. This survey reads all of those developments as one thing, contextual memory held inside the model. That framing is the survey's main move. Instead of a list of attention variants and their replacements, the field becomes a set of answers to one question: how a model should keep, and consult, what it has already seen.

To compare them, the authors introduce a lens with five dimensions. Memory Representation asks what is represented. Memory Update asks how it changes. Access asks what is eligible to be queried. Readout asks how it is read. And Integration asks how what is read out becomes the output. The lens is meant to set overlapping research lines side by side without forcing them all into one computational model. On top of it, the survey reconstructs developments at the level of mechanisms and of architectural adoption. It draws on fifty-nine release-level records from fourteen major model lineages, and on eleven open-weight endpoints that perform well.

Three threads run through what they find. First, explicit-memory methods and recurrent-state methods keep their distinct interfaces, yet increasingly control memory functions that overlap. Second, heterogeneous architectures increasingly coordinate across the depth of the network. Composition layer by layer spreads complementary memory processing over different representational stages, while reuse across layers carries selected memory and routing artifacts forward. Memory, in that picture, is no longer something each layer handles alone. Depth itself becomes an axis along which contextual memory is built and managed.

Third, these trends lead the authors to a hypothesis about stateful, multidimensional memory routing. Persistent memory, on this view, is organised across temporal scope, network depth, substrate type, and the granularity of representation. Coordinated Sparse Write and Sparse Read then decide what gets kept, and what contributes to answering each query. Writing and reading are both selective under this hypothesis, and they are coordinated with each other, so the question of what to store and the question of what to retrieve are settled together rather than apart.

Where does this leave the operator in the title? The survey's conclusion is that designing efficient sequence architectures is increasingly about how contextual memory is organised, how long it lives, and when it is selectively used. An isolated Attention operator is no longer the unit of design. The memory system around it is.