OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

OmniCapBench. A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning. By Zhongyu Yang and eleven others, from Tencent Hunyuan, Nanyang Technological University, and one other institution. Published on the eighth of October.

Multimodal large language models are moving quickly toward continuous reasoning over audio and video together, and evaluations are needed that show where their abilities run out. Audio-visual captioning makes an ideal diagnostic task. Current benchmarks for it, though, are caught in a coupled trade-off. Scoring the whole caption gives coverage but cannot say where the error is. Local probes can say where, but cover little. And leaving an LLM judge unconstrained makes the scores unstable.

OmniCapBench, short for Omni-Video Caption Benchmark, reframes caption evaluation as a deep-structured diagnostic. The prediction target is no longer free-form text. It is a set of atomic evaluation units that can each be verified, organized in three tracks: entity references, visual shots, and audio events. A caption is no longer judged as one block of prose. It is checked against a list of specific things that should be in it, each of which is either there or not.

That structure is what makes scoring reliable. Some checks are deterministic constraint checks, and the rest are LLM-based semantic comparisons kept local to a single unit. The benchmark holds seven hundred and eighty-six densely annotated videos. With them it can tell apart distinct kinds of perception error: temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions.

Frontier models show strong local perception and weak long-horizon audio-visual reasoning. The weakness is sharpest in two of those error types: identity drift, and misalignment across modalities. In plain terms, the models describe a single moment well, but over a longer stretch they lose track of who is who, and of which sound belongs with which image.

Breaking a caption into units that can be checked one at a time is what turns a single score into a map. Instead of learning that a model's caption is mediocre, one learns which kind of mistake it makes, which the authors offer as a fine-grained roadmap for omnimodal development.