Omni Interaction Agent Technical Report

Omni Interaction Agent Technical Report. By Orantqing and twenty-two others, from Tencent Hunyuan, Zhejiang University, and three other institutions. Published on the eighth of September. Its code repository is worked out: about ninety source files, with a readme and a licence.

Conventional conversational models work in turns. The user speaks, the model waits until the user has finished, then responds, and the exchange proceeds one complete turn at a time. Gander is built on a different premise. It is an end-to-end model that brings three things into a single framework: omni perception, realtime interaction, and agentic capability. Omni perception means taking in several modalities at once. Realtime interaction means responding while the exchange is still happening. Agentic capability means carrying out complex, workflow-oriented tasks, not only conversing.

Rather than taking turns, Gander receives streaming inputs continuously, across video, speech and text. That enables natural full-duplex interaction, both in everyday conversation and in complex agent scenarios organised around workflows. Users can interrupt the model at any moment. The model, in turn, can offer intermediate feedback on its own initiative, or ask follow-up questions. To support those capabilities natively, the architecture rests on two key designs. One divides the labour between fast interaction and deeper reasoning. The other determines how inputs and outputs are represented so the interaction can stay continuous.

The first is a Cerebellum-Brain collaborative framework. The Cerebellum is responsible for realtime interaction and for omni conversational ability. The Brain handles complex reasoning and higher-level agentic tasks. The two interact continuously, through tool calling and through the runtime that orchestrates the agent. Fast responsiveness and slower deliberation are assigned to different parts, which keep talking to each other.

The second design is inside the Cerebellum, which is built on a streaming Thinker-Talker architecture. User inputs and model outputs are flattened together into one ordered token stream, at the level of chunks. That gives a single unified representation for interaction that is continuous and low in latency, where listening and speaking are part of the same stream rather than separate phases.

Gander is evaluated along four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Together these cover how well it talks, how well it perceives, how well it handles the live back-and-forth, and how well it acts as an agent. Internal human evaluations show it keeping the natural, expressive spoken dialogue of state- of-the-art open-source models, while reaching competitive performance in omni interaction. It is also robust in challenging real-world conditions: background noise, conversations with several parties, and backchannel communication. Those are the conditions under which a continuously listening model is most exposed, since it cannot wait for a clean, finished turn before it acts.

The organising idea is to split interaction from deliberation without splitting the system. A part that is always listening keeps the conversation live, a part that thinks harder takes on the agentic work, and a shared stream keeps both in step with the user. The models, code and data are all released.