UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation
Deyuan Liu, Yihao Hu, Jingxuan Zhang, Xingying Li, Jun Xie, Jiacheng Liu, Jungang Li, Yu Huang and 8 others
Westlake University · Ant Group · Zhejiang University · Shanghai Innovation Institute · HKUST · CASIA
Published October 7, 2026 · on the feed October 8, 2026 · ▲ 163 · arXiv 2610.09823
The abstract
Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal different strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512's English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores. Repository: https://github.com/LINs-lab/UltraText_Bench.
The Brief
2:05A short podcast summary of the paper. Written and voiced by AI.
Transcript
UltraText Bench. A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation. By Deyuan Liu and fifteen others, from Westlake University, Ant Group, and eight other institutions. Published on the seventh of October.
Image generators have become better at rendering short strings of text. Dense visual text is a harder demand: long strings, spread across several regions, each in the right place and each legible. As the short case improves, evaluation has to move on and test whether performance holds up across more demanding scenes.
UltraText Bench is a bilingual benchmark for generating dense visual text from a prompt alone. Every generated image is assessed by the Q-Judger vision-language model against a complete reference, and four things are reported: text fidelity, text clarity, spatial quality, and scene quality.
The benchmark holds four hundred and thirty-two prompts, split equally between English and Chinese. They span twenty-four real-world scene categories and three difficulty levels. Each prompt has been reviewed by humans, and supplies the exact strings for between four and twelve text regions. Those strings come with structured references for their content, their placement, and their visual attributes, so a judge has something concrete to check each region against. Ten participants took part in a human evaluation of the automatic scores.
Twenty-four model configurations were evaluated, and the four dimensions pull apart. Z-Image-Turbo gains three point eight one clarity points over Z-Image-Base, yet loses fourteen point seven six fidelity points, under the settings reported. Clearer text, in other words, is not the same as correct text. Performance also depends on the workload. Qwen-Image-2512 scores eighty-six point five on its English composite at level one, and that falls to forty-two point eight six at level three.
Splitting the score into separate dimensions is what makes the benchmark informative. A single number would hide a model that writes crisply but gets the words wrong, and grading by difficulty level shows how quickly a strong model can lose ground once a scene asks for more text. The repository is linked from the abstract.
Paper Quest · 7 chapters · about 36 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Short text is solved, they say. Try a whole menu.
Image generators have got much better at short strings. But a menu, a receipt or a poster needs many regions of text, each correct, legible and in the right place.
The road ahead
- 1 · The problem ~7 min Image generators now draw short words much better, but menus, receipts and posters need many pieces of text that are all right at once.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Give the exact text for every region, check each region against a full reference, and report separate scores so trade-offs stay visible.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: Qwen-Image-2512 scores 86.50 on its English composite at the easiest level. What does that composite fall to at the hardest level?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.