Continuum Attention
Afshin Khadangi
University of Luxembourg
Published October 14, 2026 · on the feed October 15, 2026 · ▲ 9 · arXiv 2610.20586
The abstract
We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.
The Brief
2:04A short podcast summary of the paper. Written and voiced by AI.
Transcript
Continuum Attention. By Afshin Khadangi, from the University of Luxembourg. Published on the fourteenth of October. Its code repository is public and real: about twenty-five source files.
Standard attention looks things up. Each query is compared with a discrete set of keys, and the matching values are combined. Memory, in that picture, is a list of separate tokens. This work asks what attention looks like if memory is treated instead as something continuous.
Continuum Attention, or CA, is a causal Transformer attention operator that replaces discrete key and value lookup with integration. The memory becomes a continuous curve, one that is piecewise linear. Each query induces a relevance density along that curve, and the density is exponential. Attention is what you get by integrating the density together with the interpolated values.
The integral does not need to be approximated. It yields exact normalized attention from exponential moments in closed form, including the contributions at half-token boundaries. Gradients follow analytically, from moments of order no higher than two. That is what makes parallel GPU implementations possible. The operator keeps quadratic complexity in sequence length, and it keeps the standard Transformer decoder around it.
The comparison is against ordinary softmax attention and focal attention. CA reduces mean validation negative log likelihood by zero point zero two nats per byte against softmax, and by zero point zero four eight against focal attention, winning every paired comparison. On an additional, disjoint validation slice, it holds a zero point zero four five seven nats per byte advantage over the strongest selected softmax baseline, reported with a paired ninety-five percent confidence interval.
The author's own conclusion is measured. The results motivate continuous integration as a promising alternative for attention, and whether the advantage generalizes at larger scale is left explicitly open. What changes is the picture of memory: no longer a row of tokens to be looked up, but a curve to be integrated along, with the arithmetic still exact.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free