PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

PolyRoute. A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard. By Mohammed Nayeem Khan and one other, from Intel Corporation. Published on the tenth of November.

Typed-decision encoders answer a question about a text in a single forward pass. How they behave outside English, though, is rarely measured, and rarely against a fixed yardstick. PolyRoute is offered as that yardstick, for one concrete decision: routing a support message to one of three departments.

The release has four parts. First, three hundred and twenty-two native-language support messages, spanning twelve languages and nine writing systems. Second, one hundred and fifty-six messages that are code-mixed or romanised, in Hinglish, Spanglish and Arabizi. Third, nine hundred and sixty-seven optional native training rows. And last, a protocol for the three-way routing decision itself.

All of the texts and labels are generated by language models. Labels come from Qwen3.8-27B, with a blind second pass that kept ninety-one percent of candidates. An independent model, gemma4-31b, agrees on ninety-seven point two percent of test items. As a check on overlap, the largest character five-gram similarity between any test text and any training text is zero point three three.

A leaderboard covers thirteen systems. The English-root checkpoint scores sixty-one point two percent. The shipped multilingual checkpoint scores seventy-seven point three percent, and the best fine-tune reaches ninety-six point six percent. TF-IDF with logistic regression reaches ninety-one point nine percent.

On four withheld languages, the three multilingual systems score seventy-seven, ninety and ninety-three percent. Code-mixed text turns the order around: the English root, at eighty-five percent, is nominally ahead of the shipped multilingual checkpoint, at seventy-nine percent, with a p-value of zero point one three four.

The authors also state what their split can and cannot tell apart. It resolves differences only above about four points. That makes the large gaps on the leaderboard readable, while the English root's lead on code-mixed text stays a nominal one. And every text and every label in PolyRoute was produced by a language model.