Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Triadic Linear Attention. Three-Dimensional Recurrent States for Long-Context Sequence Modeling. By Oliver Sieberling and five others, from the Massachusetts Institute of Technology and the MIT-IBM Computing Research Lab. Published on the twenty-ninth of September. Its code repository is public and real: about twenty source files.

A recurrent network squeezes everything it has seen into a memory state of fixed size. That is what buys constant-time inference, and it is also the limit. How big that state is turns out to be a crucial factor in how well the model performs. Linear attention is the clearest example: its strong performance and its resurgence come from replacing the vector-valued hidden state of an ordinary recurrent network with a matrix-valued one.

The key detail is how linear attention gets that bigger state cheaply. It writes to the matrix using an outer product of a key vector and a value vector, so the state grows without a matching growth in parameters. Triadic linear attention takes that construction one dimension further. Instead of a matrix, the state becomes a third-order tensor, a three-dimensional block of numbers.

Writing uses a triadic outer product: a key, a second key, and a value, multiplied together into the tensor state. Reading contracts both key axes with two queries. The economics are what make this attractive. If the second key has E dimensions, the state grows E-fold, while the model gains only two extra projections. The construction also fits with the machinery modern linear attention relies on: data-dependent forgetting, the delta rule, and chunkwise-parallel training all remain compatible.

The authors apply triadic linear attention to two existing architectures, Gated DeltaNet and scalar-gated linear attention. In both, it substantially improves long-context language modeling and recall. It also outperforms alternative ways of enlarging the state. That comparison is the one that counts: if state size drives performance, then the real question is whether this particular way of growing the state beats the other ways of growing it.

A step from vectors to matrices revived linear attention; this paper asks whether the next step, from matrices to three-dimensional tensors, pays off again, and keeps the cost to two projections while doing it.