Retrieval Capacity of Self-Attention Under Competition

Retrieval Capacity of Self-Attention Under Competition. By Timur Mudarisov and two others, from the University of Luxembourg and the London Institute for Mathematical Sciences. Published on the twenty-ninth of September. No code repository is linked to it.

A language model is handed a long context. How many of those tokens does it actually use, and what decides that number? The authors answer through self-attention, and their answer has two parts. First, a relatively small set of tokens can keep the model's loss close to what it gets with full attention, though how small varies from one model to another. Second, the size of that set is not fixed by how much there is to retrieve. It grows with competition. When more background crowds the context, the model needs a larger set to stay within the same loss, even when no additional distinct information is there to find.

The measurement works without any retraining. At each head, each layer and each query, only the tokens with the highest attention weights are kept, and their original weights are left unchanged. Varying how many are kept, and watching how much the negative log-likelihood rises, gives an effective attention set size: the number of tokens needed to stay within a chosen loss tolerance.

Several observations follow from that probe. Selecting by attention substantially beats selecting at random. The selected sets show geometric structure, though the authors are careful that geometric separation alone does not establish that the model's loss is preserved. Extending the context while evaluating the same prediction targets raises the required set size, even as its fraction of the context falls over the tested range. With a fixed supporting fact planted in the context, adding background pushes that fact's tokens down the attention ranking and reduces the attention mass they receive. And renormalizing the retained weights can substantially shrink the required set, which shows the number depends not only on which tokens are selected but on how their representations are combined. Conditional theoretical models then explain how competition, together with how much attention mass is retained, can produce growing set sizes without more to retrieve.

The limits are stated plainly in the abstract, and they matter. The falling fraction of context holds over the tested range, not beyond it. The required size differs across models, so no single number is being offered as universal. The geometric structure is reported as an observation, explicitly not as proof that loss is kept. The theoretical models are conditional: they show how competition can produce the growth, under their assumptions, rather than proving that it must. And the whole quantity is defined relative to a loss tolerance the experimenter chooses, so the set size is always a size for a given tolerance.

What the work contributes is less a single answer than an instrument. It turns the question of how much context a model uses into something measurable, then shows that the measured quantity bends with three things: how long the context is, how much irrelevant material competes for attention, and how the retained pieces are added back together. The middle one carries the title's word, competition. A model can need more tokens without having more to find, simply because there is more around the thing it is looking for.