Cross-Lingual Alignment for Decoder-Only Models using MoE Routers

Cross-Lingual Alignment for Decoder-Only Models using MoE Routers. By Lucas Bandarkar and four others, from the University of California, Los Angeles, Haverford College, and two other institutions. Published on the second of October. Its code repository is public and real: about twenty-five source files.

Cross-lingual contrastive learning has been a core part of training multilingual encoders: representations of the same content in different languages are pulled explicitly together. Decoder-only language models have not been able to borrow that tool. Their multilingual tokenization varies from language to language, which makes explicit alignment of representations impossible in the form encoders used. That would matter less if alignment did not help, but a growing body of research suggests that even in large language models, higher cross-lingual representational alignment leads to better cross-lingual transfer.

The authors set out to reimagine contrastive alignment within the architectural constraints of modern models. Their move is to change what gets aligned. Rather than placing an auxiliary alignment loss on hidden states, they take the outputs of the mixture-of-experts routers as the alignment target. What the routers produce becomes the thing that two languages are asked to agree on, and the hidden states are left without a loss of their own.

The argument for routers is practical. Router outputs lend themselves better to pooling over many tokens, so a whole sequence can be summarised and compared with its counterpart in another language. That makes the cross-lingual comparison more reliable at the sequence level, rather than depending on any one token lining up with another.

The test is controlled continual pre-training on four open-source MoE models. Adding the routing loss turns out to align more than the routers: the underlying hidden representations become more aligned across languages as well. And, the result the authors call most important, the loss improves multilingual performance on their diverse evaluation suite.

The route here is indirect by design. Hidden states were the thing people wanted aligned, but they could not be compared cleanly across languages. Aligning the routers instead was enough to bring the hidden states along.