OmniEcho: Spatial Audio Understanding for Embodied Agents
Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo and 5 others
Peking University · Alibaba Token Hub, Alibaba Group · Tsinghua University
Published September 20, 2026 · on the feed September 25, 2026 · ▲ 26 · arXiv 2609.23407
The abstract
Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose OmniEcho, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges.
The Brief
2:29A short podcast summary of the paper. Written and voiced by AI.
Transcript
OmniEcho. Spatial Audio Understanding for Embodied Agents. By Ruixun Liu and twelve others, from Peking University, Alibaba Token Hub, Alibaba Group, and one other institution. Published on the twentieth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
People locate the direction of a sound without effort, and fold it together with what they see in order to reason about a scene. Embodied agents still find this hard. Underneath that difficulty the abstract names a more basic gap: it is still unclear how spatial audio understanding in embodied settings should be evaluated at all, let alone how it should be modeled.
OmniEchoBench is the answer on the evaluation side, a unified benchmark that covers two abilities. One is spatial audio-visual perception. The other is audio-vision-language navigation, where the agent is guided by sound.
The benchmark comprises six tasks built over one hundred and ninety-seven real-world spatial audio-visual scenes. Those scenes carry two thousand nine hundred and seventy-two question-answer pairs. Alongside them are nine hundred navigation samples. The audio is first-order ambisonics, or FOA, collected from thirty real-world environments.
Evaluation is only half the work. For training supervision at scale, the authors develop a controllable rendering pipeline for spatial audio, one that keeps sound sources, visual observations and agent trajectories geometrically consistent with each other. On that foundation they build OmniEcho, a spatially aware omni-modal model. What it adds is an FOA spatial encoder, placed alongside a pretrained semantic audio pathway, so a spatial route and a semantic route run side by side.
In extensive experiments, OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. On sound-guided navigation, it reaches a performance level close to that of traditional vision-language navigation. The authors take this as a demonstration that spatial audio can serve as a valuable signal for embodied scene reasoning and for navigation. The same results also mark what remains hard. Fine-grained spatial localization, and the estimation of distance, are both highlighted as important open challenges.
OmniEchoBench gives hearing where something is a test that sits alongside seeing it, in real scenes, with audio collected from real environments. Its first results point two ways at once. Sound-guided navigation reaches a level close to vision-language navigation, yet the finer questions, exactly where a source sits and how far away it is, are still open.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a robot that hears an alarm but cannot tell where it is.
You are building a home robot that sees and understands speech. An alarm goes off somewhere behind a table, out of the camera's view. A person would turn straight toward it.
The road ahead
- 1 · The problem ~7 min A robot that hears what a sound is, but not where it comes from, cannot turn toward an alarm it cannot see.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Add a spatial encoder for four-channel audio next to the kept semantic audio pathway, and train it on geometrically consistent rendered data.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Render consistent spatial training audio, graft an FOA spatial encoder beside the semantic pathway, then test on real perception and navigation tasks.
- 6 · The experiment ~6 min You’ll guess: Guided by spatial audio instead of route instructions, how did OmniEcho's navigation compare with traditional vision-language navigation?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.