EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

EvaOmni. An Evidence-Grounded Benchmark for Audio-Visual Reasoning. By Xiaoxing Hu and nineteen others, from Shanghai Jiao Tong University, the Shanghai Innovation Institute, and eight other institutions. Published on the twentieth of October. Its code repository is public but slight: five source files.

Omni LLMs, models that take in sound and vision together, can naturally locate the key audio and visual evidence in a video and integrate information across the two modalities to reason about it. The benchmarks that test this ability mostly rely on multiple-choice questions, and they usually score a model only on whether its final answer is correct.

Two problems follow. First, a correct letter says nothing about how it was reached, so it is hard to tell whether a model actually formed a chain of reasoning from the evidence to the answer. Second, textual or unimodal shortcuts can let a model answer correctly without analysing the audio-visual evidence the question was meant to require. A model might lean on the wording alone, or on only one of the two streams. Both problems weaken how reliable these benchmarks are as evaluations.

EvaOmni measures grounded reasoning rather than answers alone. Beyond a set of high-quality question-answer pairs, it requires a model to return two things at once: the answer, and the intervals of supporting evidence it relied on. That way the complete reasoning chain can be evaluated, not just its last link.

Two scores do the evaluating. The Coverage Grounded Score measures how much of the annotated evidence a model manages to identify. The Bottleneck Grounded Score penalises poor temporal localisation in either modality, so a model that pins down the audio but misplaces the visual evidence, or the reverse, is marked down. Both scores are incorporated into answer correctness rather than reported beside it.

The benchmark contains three hundred ninety-two videos and one thousand and twenty-seven question-answer pairs, all of them purely human-annotated. It covers fifteen capabilities, arranged in three levels: retrieval and localisation, basic understanding, and complex reasoning.

Construction runs through a series of rigorous procedures. Experts annotate the audio and visual evidence, and they construct distractors based on the video itself. Humans then review whether each question meets strict inspection criteria. Finally, models are used to filter out the textual and unimodal shortcuts described earlier, so that what remains should need both streams.

The evaluation covers seven open-weight and seventeen proprietary Omni LLMs. The strongest was Gemini3.7-Flash, which achieved the highest overall accuracy at seventy-nine point four percent, the highest coverage score at seventy point eight percent, and the highest bottleneck score at thirty-five point one percent.

The third level, complex reasoning, exposes the gap most sharply. There, Gemini3.7-Flash reached seventy-nine point one percent accuracy, yet its bottleneck score was only twelve point three percent.

Right answers and correctly located evidence come apart, and an answer-only score would never have shown it. The authors read their results as evidence that even the most advanced models still have room for development in grounded audio-visual reasoning.