Multilinguality in Hybrid Attention LLMs

Multilinguality in Hybrid Attention LLMs. By Lucas Bandarkar and four others, from the University of California, Los Angeles and Fudan University. Published on the twenty-eighth of September. No code repository is linked to it.

Many state-of-the-art LLMs now mix variants of attention. Agentic and reasoning use cases demand long sequences, and traditional softmax attention scales quadratically, so these hybrid models balance full attention against alternatives based on recurrence. This work presents a first study of what that hybrid design does to multilinguality. The concern is not only long sequences in poorly tokenised languages. It is that the inductive biases of the recurrent state might change how language itself is processed. The claim is that they do, and that the order of the layers matters.

An interpretability analysis confirms the suspicion. In hybrid models, cross-lingual representations develop in patterns tied to the ordering of recurrent and full-attention layers. Across diverse models, the authors observe a pronounced spike in cross-lingual alignment around the first full-attention layer.

That observation led them to question the conventional ordering of attention layers, and to test alternatives. In distillation experiments on multilingual data, every alternative layer ordering outperforms the standard one throughout training, learning up to two point five times faster. The authors describe these results as stark and replicable.

The evidence comes from interpretability analysis across diverse hybrid models, and from distillation experiments on multilingual data. The proposal that grows out of it is stated as a theory rather than an established result: that multilingual models would benefit from starting with a full-attention layer rather than with recurrent layers. The faster learning is shown under distillation, so how far the same ordering helps in other training regimes is the question the theory leaves open.

The thread running through the study is placement. Hybrid models usually decide where their full-attention layers go for reasons of sequence length and cost. This study finds that the decision also shapes how languages relate inside the model. Around the first full-attention layer is where cross-lingual alignment jumps, and so reordering the layers is the natural experiment. The distillation results say reordering helps, and helps throughout training rather than only at the end. A choice made for efficiency turns out to bear on how a model handles many languages at once.