Kronecker Delta Rule for Long Context Linear Attention

Kronecker Delta Rule for Long Context Linear Attention. By Anamaria-Roberta Hartl and five others, from Johannes Kepler University Linz and NXAI. Published on the nineteenth of October. Its code repository is worked out: about seven hundred and fifty source files, with a readme and a licence.

Recurrent linear attention architectures keep what they have read in an associative matrix memory, and the delta rule is one way to update that memory. Two concerns shape this work. The first is interference: a delta-rule update can disturb information already stored, and over a long context a great deal has been stored. The second is stability. Keeping a memory state well behaved across an entire input sequence typically calls for added normalization and stabilization mechanisms.

KronLSTM is a linear attention architecture built on the Kronecker Delta Rule, a new update rule for associative matrix memories. It extends the xLSTM family with a memory-revision mechanism. That mechanism corrects targeted associations, but modifies the memory only along a single direction in matrix space. The authors describe this as minimally invasive memory revision: it repairs the association that needs repairing while touching as little else as possible, which reduces interference with previously stored information compared with delta-rule updates. That is what makes it particularly well suited to long-context language modeling.

The memory dynamics that result can be described mathematically. They form a Markov chain that converges to a steady-state distribution, and the expected norm of the memory matrix remains uniformly bounded throughout the entire input sequence, however long it runs. Hence KronLSTM ensures stable memory dynamics without requiring additional normalization and stabilization mechanisms for the memory state.

In experiments, KronLSTM is competitive with existing linear attention architectures, and often outperforms them, including models of the DeltaNet family and of the Gated DeltaNet family. Its performance is particularly strong on long context benchmarks, which is the setting the low-interference update was meant for.

Two properties arrive together from one design choice. The restricted update that reduces interference also yields memory dynamics that can be analysed: a bounded norm, and a steady state. On that basis the authors present the Kronecker Delta Rule as an effective and analytically tractable foundation for stable associative memory in recurrent linear attention architectures.