OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

OmniVBench. A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation. By Wenxue Li and fifteen others, from Tencent's Online Video business unit and the Hong Kong University of Science and Technology, Guangzhou. Published on the eighteenth of September. No code repository is linked to it.

Reference-to-video generation is moving toward increasingly general and versatile reference control, the emerging paradigm of omni reference-to-video. Existing benchmarks fall short of it. Their test cases cover limited reference types and compositions. Their evaluation protocols mostly assess holistic consistency with the reference, and so overlook whether individual reference factors are properly preserved, disentangled, and routed. A video can look broadly faithful to its references while mixing up which reference was meant to control what. Training data is a problem too: omni data is costly to construct, so suitable resources are scarce. The paper answers both problems at once, with one resource for evaluation and one for training.

OmniVBench broadens evaluation across reference types, fine-grained control tasks, and richer compositions of references. It covers seven task families and eighteen fine-grained tasks, spanning content, motion, style, structure, narrative, and settings with multiple references. The breadth is the point. A reference might supply what appears, how it moves, how it looks, how the scene is arranged, or how the story unfolds, and a model claiming omni control has to handle each of these and their combinations.

Scoring is factor-grounded, with twelve thousand one hundred and seventy-two checklist items written for specific cases. Each item asks whether an intended reference factor is faithfully preserved, correctly disentangled and bound to its target, and properly realized according to the instruction. Those three questions line up with the three things holistic scoring overlooked. Preservation asks whether the factor survived. Disentanglement and binding ask whether it went where it was meant to go. Realization asks whether it did what the instruction said. Because the items are written for each case, the benchmark can say which reference failed and how, rather than giving a single score for the whole video.

Alongside the benchmark comes the Omni-R2V Dataset, which brings industrial- grade training resources for diverse reference-to-video tasks to the broader research community. It draws primarily on a large-scale corpus of professional video footage and contains three hundred and forty thousand processed training samples, covering diverse reference types and multi-reference compositions. Task-specific pipelines build the reference-target pairs, offering a practical and scalable recipe for constructing such data.

Extensive evaluation of advanced open-source and closed-source reference-to- video models reveals clear performance gaps on OmniVBench, across task families and across evaluation dimensions. Those gaps highlight limitations that current models still carry. The benchmark, in other words, is doing the job it was built for: separating models along the dimensions that holistic scoring would have blended together.

The shift is from asking whether a video resembles its references to asking what each reference was supposed to contribute, and whether it did. The checklist makes that question answerable item by item. The dataset then gives the community training material aimed at the gaps the benchmark exposes.