Realtime-Venus: A full-duplex interaction system with asynchronous delegation

Realtime-Venus. A full-duplex interaction system with asynchronous delegation. By Ruixiang Zhao and twenty-six others, from Ant Group and Tsinghua University. Published on the twelfth of September. Its code repository is worked out: about one hundred and twenty source files, with a readme and a licence.

Natural interaction, in digital or physical settings, needs a system that perceives continuously and responds on time. Spoken dialogue already depends on acoustic and linguistic cues. Video interaction adds another demand: the conversation has to stay grounded in a visual context that keeps changing. And a conversation cannot simply stop while a system goes away to think or to use a tool.

Realtime-Venus is a proactive, full-duplex interaction system built from two separately trained models of nine billion parameters each. Realtime-Venus-Omni handles audio-visual interaction. Realtime-Venus-Audio handles spoken interaction. Each is a complete conversational frontend, combining continuous perception, conversational control and native speech generation. What ties those together is a shared causal timeline, which holds user inputs, model outputs and delegation events on one track. Hearing, seeing, speaking and handing work off all happen against the same clock.

Around the models sits a dual-loop runtime. One loop keeps the live interaction going. The other runs background reasoning and tool execution. A component called Realtime-Venus-Harness executes tasks asynchronously while foreground interaction continues, and returns results to be integrated into the ongoing dialogue. The talking never has to wait for the working.

Both models share one post-training recipe, which combines offline understanding, proactive full-duplex trajectories, and delegation workflows.

Among the online models evaluated, Realtime-Venus-Omni scores highest on six of eight video benchmarks. These include StreamingBench at seventy point two percent, OVO-Bench at sixty-four point seven percent, and Daily-Omni at eighty-one point three percent.

Realtime-Venus-Audio was tested on eight audio understanding and spoken question answering benchmarks. It leads the compared models on MMAU at seventy-eight percent, and on MMAU-Pro at sixty-three point two percent. It also leads on Llama Questions at eighty-three point eight percent, and on Speech CMMLU at sixty-seven point eight percent. On VoiceBench AlpacaEval it matches the best score, four point eight one.

The full-duplex behavior is measured on Full-Duplex-Bench, version one point five. There, Realtime-Venus-Audio responds to seventy-five percent of user interruptions. Its continuation rate is ninety-seven percent under backchannels, eighty-eight percent under other-directed speech, and eighty-six percent under background speech. On all three of those continuation metrics, it exceeds Gemini three point one Live and GPT-4o.

Knowing when not to stop is the part of full duplex that is easy to overlook, and the continuation numbers measure it. A backchannel, someone talking to another person, noise in the room. In each case the system mostly keeps going, while still responding to three quarters of real interruptions, and it does so with background work running on a separate loop.

Realtime-Venus. A full-duplex interaction system with asynchronous delegation. By Ruixiang Zhao and twenty-six others, from Ant Group and Tsinghua University. Published on the twelfth of September. This is the long version: the whole paper rather than the abstract.

A voice assistant that can look things up has two clocks to keep. The conversation runs in real time: the user talks over it, says "right" while it speaks, turns to someone else in the room. A search or a tool call runs on its own schedule, and by the time the answer comes back the conversation may have moved on. A background task needs a stable record of what was asked and on what evidence, yet its result has to be interpreted in a conversation that may have changed while it ran. Coordinating the two is the whole design problem.

Two things to hold before the mechanism. The first is what full duplex means here: the model listens while it speaks. Both models are built by adapting MiniCPM-o 4.5, an open model of nine billion parameters, and inherit its habit of cutting time into one-second chunks. In every chunk the model reads the latest audio, plus a video frame for the Omni variant, and emits a control token: listen or speak. If it speaks, it writes a little text, which a separate speech decoder turns into sound.

The second is that there are two models, trained separately. Realtime-Venus-Omni handles audio and video; Realtime-Venus-Audio handles speech alone, and never sees the video data, about seventy percent of the corpus.

The first piece is a private channel inside the token stream. Four new tokens are added to the vocabulary, marking the start and end of a delegate span and of a backend span. When a request needs outside help, the model writes a self-contained instruction in plain language inside a delegate span. The host strips it out, so it is never shown or spoken. The model can say aloud that it will check, close its turn and go back to listening. When the result is ready, it arrives inside a backend span at a chunk boundary, as context rather than as something the model is trained to predict, and the model speaks it.

The second piece is the Realtime-Venus-Harness, which runs delegated work in three stages. Capture freezes a snapshot of the evidence available when the request began, fenced by timestamp, so that later observations cannot alter what the task was about. Dispatch uses a router that makes one model-based choice among registered capabilities, without looking at raw media: a multimodal one that answers from the frozen audio or video, a general one for open-ended plans, and user-defined skills. Each capability has a schema, and invalid arguments are rejected before anything runs. Return polishes the result into text meant to be spoken.

The division of labour is precise. The harness decides what the reply says and how it is worded. The frontend decides only when to say it. Each work item moves through queued, running, completed, delivering and delivered, and it counts as delivered only once playback of every speech chunk has been acknowledged. A request also carries a freshness deadline, and a result judged stale is dropped rather than spoken.

The Omni model adds a long-video memory that needs no training. It archives a frame only when motion-compensated prediction says the scene has changed, and retrieves stored frames by their match to the query, discounting near repeats. Of over two point eight million training samples, delegation is only about six percent.

Understanding survived the added behaviours. Among online models, the Omni variant scores best on six of eight video benchmarks, and the Audio variant leads on MMAU and MMAU-Pro. The paper itself notes what this cannot show: no controlled ablations were run, so no training component is isolated.

Overlap handling tests the thesis more directly. On Full-Duplex-Bench v1.5, the Audio model keeps talking through ninety-seven percent of backchannels, the best in the table, where GPT-4o carries on through only four percent of background speech. But it responds to only seventy-five percent of genuine interruptions, behind Joy-Duplex, GPT-4o and Gemini 3.1 Live. The authors' own reading is that strong continuation does not imply strong interruption handling.

Tool use tells the same story one layer down. On Full-Duplex-Bench v3, both frontends pick the right tools well, second and third on selection. Yet fewer than half of their episodes get every call and argument right, against sixty percent for GPT-Realtime.

The in-house delegate benchmark is the most revealing result, and the two models fail in opposite directions. The Audio model delegates ninety-two percent of requests that need outside help, but handles only thirty-nine point four percent of routine requests itself. The Omni model is the mirror image: it keeps eighty-four percent of routine requests local, but delegates only sixty-eight percent of the ones that should go out.

There is no limitations section. The conclusion lists future work instead: finer streaming chunks, a longer context window, and recovery from delayed or failed tool calls. The caveats are scattered through the results, and they are real. The delegate benchmark is internal and scores routing decisions only. Several full-duplex baselines are copied from other papers rather than rerun. The memory gain is not universal: for the base MiniCPM-o 4.5 on the shorter LongVideoBench range, memory made things slightly worse. No run-to-run variance is reported. And the machinery the title is about, frozen evidence, freshness deadlines and playback-aware delivery, is described but never measured end to end: there are no latency figures and no count of stale results.

The idea to carry away is a split between what a request means and when its answer is said. The meaning is frozen at the moment of asking, so background work cannot drift with the conversation. The timing stays live, so the conversation is never held hostage by background work. Realtime-Venus puts both on one timeline of one-second chunks, with a hidden span going out and a hidden span coming back. What the numbers say is that the frontend already listens and talks well, and that deciding what to hand off, and then completing it, is the part still unsolved.