Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

Complex KDA. Understanding and Enhancing the Expressivity of Kimi Delta Attention. By Julien Siems and ten others, from the University of Freiburg, Microsoft Research, and ten other institutions. Published on the twenty-first of September. Its code repository is worked out: about seven hundred and fifty source files, with a readme and a licence.

Linear RNNs built on the delta rule model sequences efficiently. Each update is linear with a low-rank correction, and that same structure limits what they can express. Earlier work showed that composing two delta-rule transitions inside a single recurrent update can represent a rotation in two dimensions. The price is a higher rank and more expensive updates than a single transition. The object of study here is Kimi Delta Attention, KDA, and the question is whether it can reach that rotation more cheaply.

It can. KDA can realize two-dimensional rotations by combining a single delta-rule transformation with a second reflection, and that second reflection is supplied by its channel-wise gate. Doing so requires widening the ranges KDA's parameters may take. The resulting model is called Complex KDA, or CKDA.

CKDA keeps KDA's stability and efficiency. Its transitions remain diagonal plus rank one, and they remain non-expansive. Yet it reaches the state-tracking expressivity of DeltaProduct two.

The construction pairs two ingredients KDA already has. The delta-rule transformation is one. The channel-wise gate supplies a second reflection. Combined in one transition, they yield the rotation. For this to work, two existing range extensions are combined. The gates are allowed to take values between minus one and one, and the delta-rule coefficient beta is allowed to range between zero and two.

The expressivity is then characterized precisely. The authors prove that every orthogonal matrix of the form diagonal plus rank one is exactly a CKDA transition matrix. They also show that a single CKDA layer can track every finite group isomorphic to a subgroup of SO three, and many state-tracking results need one fewer layer with CKDA than with other linear RNNs of the diagonal-plus-rank-one kind.

Two conditions frame the result. The first is structural: the transitions must stay diagonal plus rank one and non-expansive, which is what preserves KDA's stability and efficiency, and CKDA is built to stay inside that class. The second is about ranges. The result depends on both range extensions together, gates down to minus one and beta up to two, and the experiments test exactly that. Among the KDA range settings tried, combining both extensions gives the strongest length extrapolation on the S three and S four tasks and on periodic audio continuation. In language modeling, CKDA outperforms Transformers and other linear RNNs, performs similarly to a KDA baseline, and shows promising scaling behavior.

The economy of the argument is that nothing new is bolted on. The gate was already part of KDA. Widening its range, and the range of beta, lets it supply the reflection that a second delta-rule transition would otherwise have had to provide. The rotation arrives at the cost of a single transition, and the diagonal-plus-rank-one structure that keeps KDA efficient is untouched. Code and models are released openly.