OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
Haibo Wang, Jiteng Mu, Jialu Li, Jingru Yi, Yuanjun Xiong, Jianming Zhang, Lifu Huang, Mingze Xu
University of California, Davis · Adobe Research
Published October 1, 2026 · on the feed October 2, 2026 · ▲ 15 · arXiv 2610.02181
The abstract
We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. Rather than passively processing an entire audio-visual sequence in a single forward pass, OmniSeek makes evidence acquisition part of the reasoning process: it dynamically decides whether to look or listen, and over which temporal window, to retrieve sparse but critical evidence across different modalities within long contexts. Through an iterative multi-turn protocol, the retrieved raw audio or visual segments are appended back into the context to support subsequent reasoning. To cold-start this capability, we build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. We first supervise the model on these trajectories to instill multi-turn tool-use behavior, and then further optimize the policy via a two-stage reinforcement learning with verifiable rewards. Moreover, we introduce an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. Extensive experiments across a wide range of benchmarks demonstrate that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
OmniSeek. Native Tool Integration for Multi-turn Audio-Visual Reasoning. By Haibo Wang and seven others, from the University of California, Davis and Adobe Research. Published on the first of October. No code repository is linked to it.
An Omni Large Language Model typically takes in an entire audio-visual sequence in a single forward pass and reasons over all of it passively. In a long context, the evidence that actually matters can be sparse, and it can sit in the sound or in the picture. A model that only ever receives everything at once has no way to go back and check the one moment, or the one modality, that would settle a question. It can only hope the relevant detail survived being processed alongside everything else.
OmniSeek turns such a model into an active agent that reasons over multiple turns and uses tools natively. Gathering evidence becomes part of the reasoning itself. At each step the model decides whether to look or to listen, and over which temporal window, so that it retrieves the sparse but critical pieces from whichever modality holds them. The retrieved raw audio or visual segments are appended back into the context, and the next round of reasoning builds on them.
The capability has to be cold-started, so the authors build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with audio and visual evidence interleaved. The model is first supervised on these trajectories to instill multi-turn tool use. The policy is then optimised further with two stages of reinforcement learning with verifiable rewards. Supervision shows the model what good seeking looks like. Reinforcement then lets it improve on its own attempts. One more objective, called Audio-Visual Necessity, explicitly rewards successful trajectories whose reasoning depended on both modalities, which discourages shortcuts through just one of them.
Extensive experiments across a wide range of benchmarks show OmniSeek learning to seek evidence adaptively across modalities, and consistently improving audio-visual reasoning. The abstract gives no individual scores and does not name the benchmarks, so the size of the improvement has to be taken from the paper itself.
Picture a viewer who rewinds to a scene or turns up the sound on a line they missed: OmniSeek is trained toward that habit. The model decides when it has heard or seen enough, and goes back for more when it has not. Its supervision teaches the moves, reinforcement sharpens them, and the necessity reward keeps the model from settling on whichever single sense is easier.
Paper Quest · 7 chapters · about 36 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The clue is one sentence in an hour of video. A model that only watches passively may never find it.
You ask a model a question about a long vlog. The answer hangs on one remark near the end and one shot of food in the middle.
The road ahead
- 1 · The problem ~6 min A model that takes in a long video and its sound all at once can miss the one brief moment that settles a question.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Turn the model into an agent that, turn by turn, chooses to look or listen at a chosen stretch of time and reasons over what it fetches.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build example trajectories, imitate them, refine with reinforcement learning, and at run time loop: choose modality and window, append the clip.
- 6 · What they found ~3 min
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.