Omni-Streaming Thinking

Omni-Streaming Thinking. By Enjun Du and six others, from the University of Hong Kong, the Hong Kong University of Science and Technology, Guangzhou, and three other institutions. Published on the fourteenth of September. No code repository is linked to it.

A streaming omni-modal model has to decide what to answer, and when, from the video chunks and synchronized audio it has observed so far. Visual cues often support an interpretation before an utterance or a sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after the audio contradicts it. The authors call this failure premature cross-modal commitment: the eyes decide, and the ears arrive too late to change the record. Picture a moment where the image suggests one thing and the sound, a little later, says another. A system that has already written down the visual guess can go on repeating it.

Omni-Streaming Thinking, or OST, treats an interpretation as a claim to be checked rather than a fact. It generates structured outputs that include the evidence observed so far, forecasts of future evidence, and claims based on that evidence. Each claim is initially marked as pending, and linked to a future verification interval. The output is not just an answer. It is a running ledger of what has been seen, what is expected, and what is provisionally believed on that basis.

Audio evidence and visual evidence are stored separately. At the end of a claim's verification interval, OST checks the claim against the evidence from the specified modality. When contradictory evidence is detected, a refutation process reduces the influence of the claim and of the states that depend on it, and then guides a state update using the new evidence. Finally, an answer gate decides whether the claims critical to the answer meet the conditions for giving a response. Keeping the modalities apart is what makes the check possible: a claim that rests on audio is checked against what was actually heard, not against the picture that prompted it. And refutation does not only drop the claim. It also weakens whatever was built on it, and then rebuilds the state from the new evidence.

OST uses a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation. On five streaming and audio-visual benchmarks, it outperforms the strongest open baselines by more than ten percent relative, on average. The backbone stays frozen, so the gains come from the thinking structure and a lightweight adaptation, not from retraining the model.

The authors also introduce OST-DiagBench, which holds the video fixed and edits the audio, to test agreement, absence, contradiction, coexistence, and conflict between subtitles and speech. On it, OST reaches a d-prime of two point nine five, compared with at most one point three eight for open baselines, while also reducing auditory hallucinations induced by vision.

The fix works on the timing of belief. A glimpse is allowed to suggest an answer but not to settle it, and the audio gets a scheduled chance to overrule the picture before anything answer-critical is committed.