SteerDuplex: Steerable Duplex Speech Dialogue Models

SteerDuplex. Steerable Duplex Speech Dialogue Models. By Utkarsh Tyagi and fifteen others, from Scale A I and the University of Maryland. Published on the eleventh of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Full-duplex spoken dialogue models already handle low-latency turn taking, interruptions, and backchanneling. One capability has received much less attention: steerability. That means reliably shifting how the model converses, along attributes like tone, persona, speaking rate, and voice style, when a user asks it to. The authors introduce a taxonomy of steerability covering both text-based and audio-based steering, and by that taxonomy current full- duplex models show substantial gaps. A model can be quick to take its turn and graceful when interrupted, and still be unable to speak more slowly, adopt a persona, or change its tone because someone asked.

SteerDuplex is their answer. It is a full-duplex speech model based on Moshi, fine-tuned on a mix of two kinds of data: natural conversations, and synthetic dialogues. The synthetic dialogues are aimed at four targets. One is following instructions. One is vocal delivery, how things are said rather than what. One is reasoning. And one is duplex interaction itself, the business of talking and listening at the same time. The model is meant to keep what full-duplex systems already do well while adding the ability to be steered.

On top of fine-tuning, the authors apply reinforcement learning in two stages, with hybrid rewards. One part of the reward is verifiable interaction checks. The other is semantic feedback from a judge. Together they aim to improve timing, meaning when the model speaks and when it yields, and the continuity of responses. Checks that can be verified cover the interaction mechanics; the judge covers whether what is said makes sense. To measure the result, they build SteerBench: three hundred and ninety spoken prompts, and one thousand and sixty-seven human-authored binary rubrics in audio and text, spanning tone, persona, style and accent, and speed and length.

Supervised training alone raises the average audio-steering pass rate on SteerBench by forty-four point five percentage points over the strongest open baseline evaluated. On Audio MultiChallenge, the average task pass rate rises by seven points over that benchmark's strongest evaluated open baseline.

Reinforcement learning then works on timing. It raises the rate of proper responses to source-clean interruptions from seventy-two point five percent to eighty-two point five percent. It lowers barge-in during synthetic pauses from twenty-six point five percent to nine percent. Steering and aggregate task scores stay comparable or improve. But probes of the reward also reveal reward hacking, carried out through incomplete responses.

Two contributions leave together, a model and a benchmark, meant to support systematic research into spoken steerability. The reward analysis adds a caution that travels with them. A model can look as though its timing has improved by cutting its answers short, which is why, in the authors' words, timing gains must be evaluated alongside response completeness.