Training-Free Speech-Centric Omni Understanding with Frozen VLMs

Training-Free Speech-Centric Omni Understanding with Frozen VLMs. By Ankan Deria and four others, from Mohamed bin Zayed University of Artificial Intelligence. Published on the seventh of August. Its code repository is worked out: about seven hundred source files, with a readme.

Audio-visual understanding asks a model to interpret spoken content, visual events, and the temporal relationships between them, all at once. The standard route to that capability is expensive. Existing omni models typically introduce a dedicated audio encoder and rely on audio-video-text training, which couples the omni capability tightly to one specific vision-language backbone. It can also cost something: that training may weaken the visual and reasoning abilities the backbone already had. Three questions follow. Is native omni training necessary for every new backbone. Can speech-centric omni capability be added while the original backbone is preserved. And where do richer acoustic representations remain essential.

Training-Free Omni, or TFO, answers the second question by construction. It is a plug-and-play framework converting a frozen vision-language model into a speech-centric omni model, with no architectural modification and no multimodal re-alignment. Nothing about the backbone changes, which means nothing about the backbone can be damaged.

The routing is deliberately plain. Whisper extracts transcripts that are timestamped and filtered by confidence, and those transcripts are routed through the language interface the vision-language model already has. The visual pathway is left entirely unchanged. Speech becomes language before it reaches the model, so the model needs no new way of receiving it, and the timestamps carry the temporal relationship the task depends on.

The comparison is matched against native omni models, across fifty-six benchmarks and twenty-one languages. TFO is competitive on audio-visual understanding and achieves substantial multilingual speech gains. Average audio-only performance improves across all five model settings, which is every setting tested rather than a majority of them. The other half of the result is about what was not lost. Freezing the backbone generally preserves stronger image and video understanding, visual grounding, coding, mathematical reasoning and medical question answering than the corresponding native omni checkpoints. Both halves are needed, because parity on the new capability means little if the old ones eroded to buy it.

What the results argue is that modular audio-to-language routing can often stand in for costly backbone-specific training. Strong speech-centric omni understanding did not require rebuilding the model around audio. It required handing the model audio in a form it already knew how to read.

Training-Free Speech-Centric Omni Understanding with Frozen VLMs. By Ankan Deria and four others, from Mohamed bin Zayed University of Artificial Intelligence. Published on the seventh of August. This is the long version: the whole paper rather than the abstract.

The way you build an omni model is settled practice. Take a vision-language model, bolt on a dedicated audio encoder, and align the audio representations with the visual and language ones through large-scale multimodal training. It works. The paper's question is whether it is necessary, and the reason nobody knew is methodological rather than technical. Existing evaluations demonstrate that native omni models are effective, but they never ran the obvious control: the same backbone, the same benchmarks, and audio supplied by the cheapest possible route. Without that comparison you cannot tell what the learned audio pathway is actually buying.

Three costs motivate asking. Native training is expensive and brittle, because each new and better backbone does not automatically inherit the audio pathway, which has to be adapted and realigned again. Audio is genuinely hard to integrate, being temporally dense and often noisy, and prior work shows that even natively trained audio-visual models sometimes overlook relevant audio, or infer sounds from visual cues instead of hearing them. And joint training can weaken what the backbone already did well: image and video understanding, visual grounding, coding, mathematical reasoning, domain knowledge. Native omni training has to acquire hearing without losing anything else.

The control the authors build is deliberately plain. Whisper transcribes the audio and returns transcripts carrying a language tag, a confidence value and timestamps. The transcript, the visual content and the user's question are assembled into a single prompt and handed to the frozen model through the language interface it already has. The visual pathway is untouched, no parameters change, and nothing is realigned. Two details do real work. Timestamps are retained so that spoken segments stay associated with the visual events they accompany, which is what makes temporal questions answerable. And confidence filtering discards unreliable transcripts, with the rule that if no segment clears the threshold, the audio context is omitted entirely rather than passed along as noise.

The scope is five matched comparisons across four model families, spanning fifty-six benchmark datasets and twenty-one languages. On audio-visual understanding the training-free version wins the average in three of the five comparisons and loses two. The wins are not marginal: at the smallest scale it goes from forty-three point two to forty-six point two, and on individual benchmarks the gaps run wider still. The losses are not marginal either, and they are the informative half. The second half of the result concerns preservation: freezing the backbone generally retains stronger image and video understanding, visual grounding, coding, mathematical reasoning and medical question answering than the corresponding natively trained checkpoint. That second measurement is one existing omni studies rarely make at all: they evaluate the audio ability that was gained, and not the visual and reasoning ability that adaptation may have cost. Measuring both sides is what turns a comparison into a trade-off.

The authors state the boundary rather than working around it. Transcript-based routing cannot preserve acoustic evidence it never encoded, so tasks involving music, environmental sounds and other non-speech cues remain out of reach. There is also a practical cost the accuracy tables do not show: speech recognition happens before the model is called, so the approach adds latency that a native model with an integrated audio encoder does not pay. And the framing is speech-centric throughout, which is a scope declaration rather than an oversight.

The lasting contribution here is the control itself. A field that only ever measures its trained systems against each other cannot tell how much of the result came from the training. Establishing language-level audio routing as a strong baseline reframes the question from whether native omni training works to where it earns its cost, and the answer turns out to be specific: it earns it where the sound is not speech.