OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

OmniCompass. A Holistic Benchmark for Omni-Modal Video Grounding and Understanding. By Xuyang Liu and seven others, from the Hong Kong Polytechnic University, the OPPO Research Institute, and three other institutions. Published on the thirteenth of October. Its code repository is public and real: fifteen source files.

Multimodal large language models are developing quickly, and judging how well they understand video has become correspondingly more important. The trouble is coverage. Existing video understanding benchmarks each look at only part of the evaluation dimensions, and a partial view makes it hard to diagnose, in a systematic way, what an omni-modal model does and does not understand when it watches and listens to a video.

OmniCompass is built to be holistic about that. It asks three complementary questions of every target. Temporal grounding asks when a referenced target appears. Spatial grounding asks where it is. Question answering asks what happens. Those three tasks are posed in two separate evaluation tracks. OmniCompass-VL assesses vision-language models on tasks that require visual evidence. OmniCompass-AV assesses omni-modal models on tasks that require joint audio-visual evidence, where seeing alone is not enough.

The question answering task is subdivided by category. In the vision-language track it covers five perception categories and three reasoning categories. In the audio-visual track it covers three perception categories and two reasoning categories.

On scale, OmniCompass-VL holds eight hundred and twenty videos and two thousand four hundred and sixty instances, spread across domains and across three duration types: short, medium and long. A subset rich in narrative and audio forms OmniCompass-AV, with two hundred and forty videos and seven hundred and twenty instances. One further design choice matters for what the results can show. Each instance is evaluated under two paired settings, text only and text plus image, which specify the same target once through text and once through visual-language guidance. A model that truly understands the target should treat both the same. Every instance is manually annotated, with multiple rounds of quality verification.

The benchmark was used to evaluate fifty multimodal models. Open-source models still lag behind proprietary ones, and the gap is widest in audio-visual understanding. Precise grounding, in time and in space, remains challenging even for the leading models. And the paired settings expose something a single phrasing would hide: predictions are still inconsistent across the two specifications of the same target, so changing how a target is pointed to can change the answer.

What OmniCompass contributes is a single frame in which when, where and what are asked of the same video, for sight alone and for sight with sound. Its results point to grounding as the persistent weak spot, and to the paired specifications as a check on whether a model's answer depends on the target or merely on how it was described.