OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

OmniReasoning. Pushing the Limits of Audio-Visual Joint Reasoning. By Junming Lin and thirteen others, from Peking University and Alibaba Group. Published on the thirtieth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Unified omni-modal models now understand audio, vision and language within one system. The work around them has not caught up. Benchmarks, training data and learning methods largely treat each modality on its own. As a result, audio-visual joint reasoning, where an answer needs what is heard and what is seen together, is poorly evaluated and insufficiently elicited. A model can look capable on audio and capable on vision, and still never be tested on combining the two, let alone trained to.

The response comes in three parts: a benchmark, a data engine, and a learning method. Each one targets a piece of that gap. The benchmark measures joint reasoning, the engine produces training data that demands it, and the learning method assigns credit to clues according to the modality they come from.

OmniReasoningBench is built so that both audio and visual evidence are indispensable. It holds one thousand one hundred and fifty multiple-choice and open-ended questions, across two tasks: reasoning over video, and reasoning beyond video.

The data engine, OmniQA, automatically constructs question and answer pairs grounded in evidence, which explicitly require joint audio-visual reasoning. With them come time-stamped chains of clues that guide the annotation of the thinking process. Beyond the benchmark, this engine yields two training sets, OmniReasoning-SFT-112K and OmniReasoning-RL-19K.

The learning method is Modality-Factored Self-Distillation, or MFSD, an on-policy self-distillation approach. It evaluates each sampled response under clue contexts specific to one modality. That disentangles the contribution of individual clues from their cross-modal interactions, and the result is used for credit assignment at the level of single tokens. The aim is that each part of a response is credited according to the individual clues, and the interplay between modalities, that actually stand behind it.

Trained with this data and method, the resulting model, OmniReasoning-30B-A3B, reaches fifty percent on OmniVideoBench and forty-two point five percent on OmniReasoningBench. Measured against its base model, Qwen3-Omni-30B-A3B-Thinking, that is a gain of twelve point eight percentage points on the first benchmark, and nine point three on the second. The authors also report substantial gains on general and long-video benchmarks, Video-MME-v2 among them.

Rather than a single fix, this is an attempt to supply the whole loop at once. A test that cannot be passed with one modality alone, data that forces both to be used, and a training signal that tells clues apart by where they came from. The authors frame it as a step toward future research in omni-modal joint reasoning.