VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

VTR-Bench. A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation. By Yu Huang and ten others, from the City University of Hong Kong, the Hong Kong University of Science and Technology, Guangzhou, and three other institutions. Published on the first of October. Its code repository is worked out: about sixty source files, with a readme and a licence.

Recent video generation models produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Evaluation benchmarks have mostly assessed visual quality, aesthetic appeal, and physical plausibility. Text has received limited attention, though it is an essential medium for conveying information in everyday scenes. A generated video can look compelling and feature lifelike subjects, and still render the writing inside the scene incorrectly. Nothing in a visual quality score would reveal that. A sign with a misspelled word, or a slide whose label dissolves into shapes, can still look beautiful. Yet the moment a viewer tries to read it, the video has failed at the one job the text was there to do.

VTR-Bench evaluates the Visual Text Rendering capabilities of video generation models. It does not treat text in isolation. It situates text within concrete application scenarios, such as advertisements and scientific videos, so a model has to get the words right while also producing the scene and motion the prompt asks for. Tying the text to a scenario matters because in practice the two are never separate. An advertisement is text in a setting, moving, and a scientific video is the same. A benchmark that scored words on a blank background would miss the conditions under which the words actually appear.

The benchmark has three hundred carefully constructed prompts, spanning five scenario categories. Evaluation is automated, through a pipeline with human alignments, and it assesses two things separately. Text fidelity is checked through carrier-specific transcription. Scene and motion requirements are checked through a prompt-specific chain of query. Keeping the two scores apart means a model cannot hide bad spelling behind a good scene, or a wrong scene behind correct spelling.

The authors go beyond evaluation as well. They introduce a Keyframe-Guided Agentic Framework, in which a Director agent coordinates image and video generation with visual evaluation. Visual feedback guides iterative refinement and the selection among candidates.

Eleven state-of-the-art models were tested, and difficulty in rendering scene text accurately is widespread. Even the best-performing model records an overall word error rate of zero point two five. Put another way, for the strongest system tested, roughly one word in four still comes out wrong by that measure. The authors further analyse the text rendering failures, to characterise the challenges current video generation models face, rather than leaving the result as a single error rate.

Two conclusions sit side by side. Visual text rendering is a key challenge for video generation, one that general quality benchmarks were not set up to see. And the agentic framework, with a Director checking keyframes and refining candidates, is offered as a practical path toward improvement rather than only a diagnosis.