StepAudio 3 Realtime Technical Report
Bin Lin, Bo Zhao, Boyang Zhang, Boyong Wu, Chao Yan, Chen Geng, Chen Wu, Cheng Yi and 82 others
StepFun-Audio Team
Published September 12, 2026 · on the feed September 16, 2026 · ▲ 126 · arXiv 2609.14005
The abstract
Realtime spoken interaction demands deep reasoning, prompt responses, and fluid turn-taking. We present StepAudio 3 Realtime, an audio-language foundation model organized around a continuous listen-converse-think-act loop. Deep Perception captures rich acoustic cues to interpret user intent, while Seamless Duplex models synchronized audio streams to handle pauses, backchannels, and interruptions naturally. Crucially, we resolve the tension between deep deliberation and latency via Think-While-Speaking, executing private reasoning in parallel with spoken delivery. In reasoning mode, StepAudio 3 reaches a 73.0 macro average on StepAudioChat. With Think-While-Speaking, it achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time. Furthermore, an integrated Voice Agent handles asynchronous tool execution without disrupting the dialogue flow. StepAudio 3 Realtime achieves top-tier performance across key dimensions: an exceptional 90.6 on the MMSU benchmark, 98.9 Overall on the Artificial Analysis Full-Duplex Bench, and a 56.0% macro task-success rate on τ-Voice.
The Brief
1:51A short podcast summary of the paper. Written and voiced by AI.
Transcript
StepAudio 3 Realtime Technical Report. By Bin Lin and eighty-nine others, from the StepFun Audio Team. Published on the twelfth of September. No code repository is linked to it.
Realtime spoken interaction pulls in three directions at once. It needs deep reasoning, prompt responses, and fluid turn-taking. The first of those tends to fight the second. A model that stops to think leaves the user waiting in silence, and a model that answers at once has not thought.
StepAudio 3 Realtime is an audio-language foundation model organized around a continuous loop of listening, conversing, thinking and acting. Its answer to the tension between deliberation and latency is called Think-While-Speaking. Private reasoning runs in parallel with spoken delivery, rather than having to finish before the model opens its mouth.
Three further components fill out the loop. Deep Perception captures rich acoustic cues to interpret what the user intends, beyond the words alone. Seamless Duplex models synchronized audio streams, so that pauses, backchannels and interruptions are handled naturally rather than breaking the exchange. And an integrated Voice Agent executes tools asynchronously, without disrupting the flow of the dialogue while it waits on them.
In reasoning mode, the model reaches a seventy-three point zero macro average on StepAudioChat. With Think-While-Speaking switched on, its dialogue and reasoning performance are comparable to dedicated reasoning models, while it keeps speaking in real time. It scores ninety point six on the MMSU benchmark, and ninety-eight point nine overall on the Artificial Analysis Full-Duplex Bench. On tau-Voice, its macro task-success rate is fifty-six percent.
The central move is to stop treating thinking and talking as steps in sequence. Once reasoning runs alongside speech, the latency cost of deliberation no longer has to be paid in silence, and the conversation can keep moving while the model works.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Talking is fast; thinking is slow. This quest starts where those two collide.
You are building a spoken assistant that people talk to in real time. It must reason carefully on hard requests, answer promptly, and handle turn-taking: pauses, quick acknowledgments like 'mm-hm', and real interruptions.
The road ahead
- 1 · The problem ~7 min A spoken assistant must think hard, answer fast and handle turn-taking all at once, and those demands pull against each other.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min StepAudio 3 Realtime runs listening, conversing, thinking and acting as one continuous loop, so reasoning, turn-taking and tools never force the conversation to stop.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The model reads acoustic cues, manages the floor from both audio streams, reasons while it speaks, and runs tools in the background.
- 6 · The experiment ~6 min You’ll guess: With Think-While-Speaking switched on, how did the model's dialogue and reasoning performance compare with dedicated reasoning models that do not speak in real time?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.