VibeVoice-ASR-Streaming Technical Report
Yujie Tu, Zhiliang Peng, Jianwei Yu, Li Dong, Songchen Xu, Yaoyao Chang, Wenhui Wang, Zilong Wang and 5 others
Microsoft Research · University of Chinese Academy of Sciences · Shanghai Jiao Tong University
Published September 2, 2026 · on the feed September 3, 2026 · ▲ 13 · arXiv 2609.02812
The abstract
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
The Brief
1:53A short podcast summary of the paper. Written and voiced by AI.
Transcript
VibeVoice-ASR-Streaming Technical Report. By Yujie Tu and twelve others, from Microsoft Research, the University of Chinese Academy of Sciences, and one other institution. Published on the second of September. Its code repository is worked out: about forty-five source files, with a readme and a licence.
Speaker-attributed speech recognition has to answer two questions at once: what was said, and who said it. Traditional systems handled these as two separate tasks, recognition on one side and speaker diarization on the other. End-to-end models such as VibeVoice-ASR have since unified the two within a single model. Those unified models, however, still mainly work offline. That makes them a poor fit for real-time voice assistants and agents, which need low latency.
VibeVoice-ASR-Streaming is presented as one of the first end-to-end approaches built on a large language model to do speaker-attributed recognition in streaming form. The aim is a model that produces who said what while the speech is still arriving, with no separate diarization stage.
The input is interleaved. Fixed-size chunks of audio alternate with a small amount of lookahead audio and with the text produced so far. Each step therefore sees the newest stretch of speech, a short glimpse of what comes next, and its own earlier transcript, which is what lets it keep going as audio streams in.
For transcription accuracy, the seven-billion-parameter model achieves the lowest average word and character error rate across five evaluation sets. For speaker attribution, it is best or tied for best on twelve of thirteen evaluation settings. The team releases weights for a one-point-five-billion model and for the seven-billion model, together with inference code.
The step being taken is from offline to live. Unification had already removed the separate diarization stage. This report keeps that single-model design and makes it work on speech as it arrives, which is the condition a real-time assistant actually operates under.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A meeting, several voices, and an agent that must know who is talking before they stop.
You are building a voice agent that joins meetings with several people. It needs to know who said what while they are still talking, so it can respond quickly.
The road ahead
- 1 · The problem ~7 min A live meeting agent must know who said what while people are still talking, and existing systems are either split in two or far too slow.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Feed audio in fixed-size chunks, each with a short peek ahead, and have one model write speaker-labelled text after each chunk while remembering everything so far.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The method loops: cut audio into chunks, add a short lookahead, write who said what, keep that text in context, then take the next chunk.
- 6 · The experiment ~6 min You’ll guess: Across 13 speaker-attribution settings, how often was the streaming 7B model best or tied for best?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.