OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

OmniVChat. Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue. By Haolin He and seventeen others, from the Chinese University of Hong Kong, Alibaba Token Hub, Alibaba Group, and three other institutions. Published on the eighteenth of September. Its code repository is public but slight: three source files.

OmniVChat, short for Omni Video Chat, is defined as native audio-visual dialogue between a user and an omni model. The model receives the user's audio and video directly and simultaneously, and replies in text. The query lives inside the audio and video themselves, with no separate text question, no external captioning, and no speech recognition. Taking audio-visual input directly reduces external latency and computation, and it keeps perceptual cues that a transcript would lose.

Research on this task meets two constraints. The first is data: recordings of people using their own devices are scarce. The second is evaluation. A good reply often has to account for the user's surroundings, facial expressions, and nearby objects, and such a reply can be expressed in many different ways. Keyword matching is therefore unreliable for judging reply quality, because two good answers may share almost no words.

OmniVChat-Bench is the evaluation built for this task. It measures an omni model's basic dialogue abilities across five ability categories. It is constructed from synthesized dialogues, and it has a human-recorded companion set, OmniVChat-Bench-Human, which serves as a check against real people. The five categories cover basic dialogue ability, the foundation a native audio- visual conversation has to stand on.

The synthesis rests on one premise. Recent progress in agent systems and video generation makes what the authors call generation for comprehension viable: using synthesized dialogues for training and evaluation. OmniVChat-Studio is the resulting multi-agent data engine, producing both single-turn and multi- turn audio-visual dialogues. For training, OmniVChat-RL is a reward design for reinforcement learning that jointly targets three qualities of a reply: its correctness, its efficiency, and its style. The three pieces fit together as a pipeline. The studio generates dialogues. The benchmark, built from them, evaluates. The reward design, trained on them, improves a model. Each answers one of the two constraints: synthesized data for scarcity, and a benchmark built for replies that can be phrased many ways.

Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on OmniVChat-Bench, and also on the human-recorded OmniVChat-Bench-Human. The authors read these gains in two ways. They validate the reward design. And they show transfer to real-world dialogue, both in training and in evaluation, since a model trained on synthetic conversations improves on recorded ones.

One move runs through the whole contribution. Where real recordings are scarce, generate them, and use them both to teach and to test. The human- recorded set is what keeps that move honest, and it is where the reported gains carry over.