SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?

SleuthVLM. Do We Really Need Audio-in-Video for Omni Video Understanding?. By Peiran Wu and eight others, from the University of Bristol, Ant Group, and one other institution. Published on the fourth of November.

Omni models answer questions about a video by processing its sampled frames and its audio jointly, in a single turn. Progress on them has been driven largely by scale: larger models, and more omni training data. Underneath that recipe sits an assumption, which is that understanding what a video sounds like means the model has to hear it. The authors put the assumption to the test with a blunt question. Does a model need to hear the audio directly at all?

SleuthVLM is their answer, and the answer is no. It is an agent harness built on a vision-language model, a VLM, that never hears the audio directly. Speech is transcribed into text, and that text is what the VLM works with. From there the model sleuths the video on its own, with no predefined workflow telling it what to examine or in what order. The name carries the picture: a detective working a case, rather than a listener taking in a recording in one pass.

The harness divides the work along a simple line. Hearing is reduced to transcription, and everything after that belongs to the agent. The authors draw the contrast in their own terms. An omni model glances at sampled frames and audio in one turn, while SleuthVLM investigates. The design point the abstract stresses is the absence of a scripted procedure: the agent decides how to sleuth.

Testing covers four omni benchmarks, with videos up to ninety minutes long. Across them, SleuthVLM achieves the best average accuracy, seventy-five point zero percent, which is two point six points above Gemini three point eight Flash. It also tops the Video-MME-v2 leaderboard, at seventy-seven point eight percent. The result does not hinge on a closed model underneath either. Built on an open-source VLM of twenty-seven billion parameters, it beats all open-source omni models.

Three findings are drawn from this. SleuthVLM beats a model that glances. Hearing is not the bottleneck; agentic ability is. And omni understanding needs better agentic ability, not just more omni data for modality alignment. The authors offer the harness as a simple but effective starting point, and compress the argument into a single line: better agents, not just bigger models that hear.