All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin and 1 others
Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Tencent Inc. · South China University of Technology
Published September 21, 2026 · on the feed September 23, 2026 · ▲ 55 · arXiv 2609.24058
The abstract
Multilingual scene text recognition (STR) remains challenging due to the scarcity of training data for most languages and the difficulty of serving diverse scripts within a single model. Existing solutions either deploy one recognizer per language, inflating cost and introducing error accumulation, or rely on massive vision-language models (VLMs) that are expensive and still inaccurate on many scripts. In this work, we pursue an all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. First, we construct TextMuSS-10M, a large-scale synthetic scene text dataset spanning 10 scripts and 229 languages. It provides balanced and sufficient supervision where real data is unavailable. Second, we propose ScriptMoE, a script-aware Mixture-of-Experts (MoE) architecture. It shares a single visual encoder and replaces the dense decoder with a sparse MoE block, which consists of an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert absorbs cross-script knowledge. Extensive experiments on our assembled TextMuSS-Bench (10 scripts, 10,899 images) show that ScriptMoE achieves the highest accuracy of 82.06%, outperforming the strongest STR baseline by 1.31%. On the CC-OCR end-to-end multilingual task, replacing only the recognizer in PP-OCRv5 with ScriptMoE lifts F1 score from 65.71% to 80.89%, slightly surpassing the best VLM (80.73%) at a fraction of the parameter count.
The Brief
2:46A short podcast summary of the paper. Written and voiced by AI.
Transcript
All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts. By Xingsong Ye and eight others, from Fudan University, the Shanghai Key Laboratory of Multimodal Embodied AI, and two other institutions. Published on the twenty-first of September. Its code repository is worked out: about two hundred and fifty source files, with a readme.
Multilingual scene text recognition, or STR, is hard for two reasons. Most languages have very little training data, and one model has to serve many different scripts. Existing solutions go one of two ways. Some deploy a separate recognizer per language, which inflates cost and lets errors accumulate across the pipeline. Others rely on massive vision-language models, or VLMs, which are expensive and still inaccurate on many scripts. Neither route gives a single model that is both cheap and accurate across scripts, and that is the gap this work aims at.
The goal here is a single all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. It has two parts: a dataset to fill the data gap, and an architecture to handle the variety of scripts.
The dataset is TextMuSS-10M, a large synthetic scene text collection spanning ten scripts and two hundred and twenty-nine languages. It supplies balanced and sufficient supervision where real data does not exist. That answers the first difficulty directly. Languages with little real training data get synthetic data instead, in balanced amounts, so no script is starved.
The architecture is ScriptMoE, a script-aware Mixture-of-Experts. It shares one visual encoder across everything, and replaces the dense decoder with a sparse MoE block. Inside that block, an image-level router sends each image to the top two experts aligned with scripts, while a shared expert absorbs knowledge that crosses scripts. The model can specialise by script without splitting into separate recognizers.
The evaluation uses TextMuSS-Bench, assembled by the authors: ten scripts and ten thousand eight hundred and ninety-nine images. ScriptMoE reaches the highest accuracy, eighty-two point zero six percent, beating the strongest STR baseline by one point three one percent.
The end-to-end test is on the CC-OCR multilingual task. Replacing only the recognizer inside PP-OCRv5 with ScriptMoE lifts the F1 score from sixty-five point seven one percent to eighty point eight nine percent. That slightly surpasses the best VLM, at eighty point seven three percent, with a fraction of the parameter count.
The claim is that one model can cover many scripts, if the data are balanced and the model is allowed to route by script. The end-to-end result makes the practical case: swap one component into an existing pipeline, and it reaches the best VLM's level at a fraction of the size.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Hundreds of languages, almost no data for most, and one model to read them all.
You need to read text in photos across many writing systems: Latin, Arabic, Thai, Tibetan and more. Most of them have almost no real training images.
The road ahead
- 1 · The problem ~7 min Reading words in photos across many writing systems is hard because most scripts lack real training images and current systems are either fragile chains or huge models.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min One recognizer shares a visual encoder across all scripts, routes each image once to script-aligned experts, and keeps a shared expert for common knowledge.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Synthesize balanced data for ten scripts, encode each image once, route it to two script experts, and mix in an always-on shared expert.
- 6 · The experiment ~6 min You’ll guess: In an existing OCR system, only the text recognizer was swapped for ScriptMoE. How did its end-to-end F1 compare with the best VLM?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.