Multilingual GSM-Symbolic: What determines capability transfer across languages?

Multilingual GSM-Symbolic. What determines capability transfer across languages?. By Kenneth Enevoldsen and twenty-four others, from Aarhus University, Danish Foundation Models, and sixteen other institutions. Published on the second of October. Its code repository is worked out: about forty-five source files, with a readme and a licence.

Very little is understood about how a capability a model acquires in one language carries over to another, or about what governs that transfer. Part of the reason is the evaluations. They rely on datasets that cannot be compared with one another and that tend to saturate, and they rarely examine the possible determinants of transfer together. If one could identify what predicts transfer, two things would follow. Nobody would need to evaluate exhaustively across every pair of languages, and developers could target the specific factors that hold performance back in low-resource languages.

The instrument built for this is Multilingual GSM-Symbolic, an extensible mathematical dataset of thirty thousand question and answer pairs, matched item for item across fifteen languages. Because every language carries the same items, a difference in score can be read as a difference in transfer rather than a difference in test. The quantity being measured is how much of a model's mathematical capability survives the move from one language to another, and the question the title asks is which properties of the model and of the language decide how much survives.

The questions come from symbolic templates. A single sample can generate millions of high-quality variations, which is meant to prevent overfitting and to ensure that what gets measured generalises. On top of the dataset sits an analysis framework that estimates the candidate determinants jointly, in one model, instead of one at a time.

The largest determinants of capability come out as model size, with a coefficient of one point seven seven, language resource level at zero point seven seven, reasoning at zero point six seven, and typological distance at minus zero point two five. Size matters most, how well resourced a language is comes next, and typological distance is the one factor whose sign is negative: the more distant the language, the lower the expected score.

Estimating them jointly lets one be expressed in terms of another. A thirty-two billion parameter model evaluated in Marathi performs like a ten billion parameter model in English. Model size and reasoning also narrow the gap between low- and high-resource languages, while similar levers do little or nothing for typologically distant languages. The framework explains ninety-two percent of the variation between languages, but only twenty-three percent of the model by language variation. It predicts a model's score on an unseen language to within six percentage points, with a correlation of zero point nine six, and adding measurements from just ten templates in the target language brings that error down to four point one nine points.

What this offers developers is an exchange rate. Language, size and reasoning can be priced against each other, and a handful of templates in a new language is enough to estimate performance there without building a full downstream dataset. The caveat the authors give is in their own numbers: most of the variation between languages is explained, but much of what is specific to one model in one language is not.