Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents
Yanjie Zhang, Nanchen Hu, Yushi Sun
HKUST · LIGHTSPEED
Published September 26, 2026 · on the feed October 2, 2026 · ▲ 11 · arXiv 2609.32536
The abstract
Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VoxGate as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity.
The Brief
2:16A short podcast summary of the paper. Written and voiced by AI.
Transcript
Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents. By Yanjie Zhang and two others, from the Hong Kong University of Science and Technology and Lightspeed. Published on the twenty-sixth of September. No code repository is linked to it.
Audio language models can recognize a spoken command and invoke a tool for it. Before doing that, though, a voice agent has a prior decision to make: does the acoustic and conversational context actually call for action at all? A command overheard is not a command addressed to you. That decision, whether to act, is what this work sets out to diagnose.
VGBench is a diagnostic benchmark of one thousand and eighteen items for action-level addressedness. It spans three kinds of scenario: side-talk, self-talk, and speaker switches. Every item shares one action space with three options. The agent can stay silent, make a tool call, or answer in natural language.
The speaker-switch pairs carry the sharpest control. The words spoken are held fixed. What changes is the source, the distance rendering, and a temporal boundary, which together define a controlled shift from the wearer of the device to a bystander. Factorized controls then pull those cues apart, so their separate effects can be seen.
Six raw Audio LLMs and three training-free adaptations often pick out the right tool, yet rarely hold back from acting once the speaker has switched. The best raw switch mute rate is fourteen percent.
VoxGate serves as a post-training case study. Supervised training mutes ninety-one point three percent of switched commands while still choosing the correct tool for every nearby wearer command and every text-only control. An exploratory GRPO stage gives similar switch performance. Side-talk accuracy rises from sixty-eight point four to seventy point nine percent, and self-talk muting from fifty-two to sixty percent. The factorized controls find an independent effect of the source changing, while sensitivity to the far-field manipulation varies from one acoustic rendering to another.
Those controls settle what the benchmark is really testing. A model passing it is not simply recognizing who is speaking. It is weighing several acoustic cues together to decide whether to act, which the authors call multi-cue acoustic-context gating. Off the shelf, the models mostly act anyway.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is an agent that acts on any voice it hears.
You build a voice agent that can send messages and place orders. A bystander says the same command your user might, or your user mutters one while thinking aloud.
The road ahead
- 1 · The problem ~7 min A voice agent that hears a command must first decide whether the command was meant for it, and most current tests do not check that decision.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Test whether context controls action by keeping the words fixed, changing who speaks, and scoring silence together with correct tool use.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build word-matched pairs, test existing models on them, then train a model with supervised learning and an exploratory GRPO stage.
- 6 · The experiment ~6 min You’ll guess: Among six raw Audio LLMs, what was the highest rate of staying silent when the same command came from a bystander instead of the wearer?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.