Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning

Building Multilingual Bridges. Data Mixing as the Pillar of Generalization for In-Language Reasoning. By Mehrnaz Mofakhami and seven others, from Cohere Labs and Cohere. Published on the ninth of September. No code repository is linked to it.

Reasoning language models have advanced on many complex tasks, but their reasoning is overwhelmingly English-centric. Whatever language they are prompted in, they mostly reason in English. That is inaccessible to users who do not speak English, it risks losing the intent of the original question, and it gives up knowledge that is more readily expressed in the target language. The authors pursue what they call L2 reasoning: a model reasoning consistently in the language of the user's prompt, building an in-language bridge from prompt to answer.

Their central claim is that data mixing is the pillar of generalization here. Reasoning, they conclude, is a language-agnostic behaviour. It can be transferred across typologically diverse languages through careful data mixing, without needing reasoning supervision in every target language. Put plainly, a model does not have to be shown how to reason separately in each target language, one language at a time. It has to be shown how to reason, and separately shown the languages.

The approach is data-centric: it studies how to optimize the composition and scheduling of data in supervised fine-tuning, SFT, so that reasoning generalizes. The result is Tiny Aya L2-Thinker, a model at the scale of three point three five billion parameters. It reasons in the prompt's language at a rate above ninety-three percent, across sixty languages and six benchmarks, while keeping performance strong.

Those benchmarks span math, commonsense reasoning, instruction following, open-ended generation, and cultural reasoning. On the question of reaching languages held out of training, the authors identify three ingredients. Broader language coverage. Multilingual data that is not about reasoning but is readily available. And a sufficient backbone of reasoning data in English. None of the three requires writing reasoning data in every language a model is meant to serve. Each is either easy to find or already exists, which is what makes the path practical for languages beyond the ones trained on.

The evidence comes from one model at one scale, trained with SFT, and the rate reported is the rate at which it reasons in the prompt's language, measured on the benchmarks named. The claim about transfer is specifically that per-language reasoning supervision is not required, not that the data mix does not matter: careful mixing is the condition. The model weights and the multilingual reasoning data are released for further research.

The recipe that emerges is a division of labour among data types. English reasoning data supplies the reasoning. Broad multilingual data, even without reasoning in it, supplies the languages. Mixed with care, the two combine into a model that thinks in the language it was asked in, including languages it never saw reasoning examples for.