WorldSonus: Bringing Sound to Worlds

WorldSonus. Bringing Sound to Worlds. By Pengjun Fang and eleven others, from the Hong Kong University of Science and Technology, Noiz AI, and two other institutions. Published on the sixth of October.

World models now produce increasingly realistic visuals, but the environments they generate are largely silent. Adding sound is harder than attaching an audio track afterwards, because three demands arrive together. Generation has to be real-time, to keep pace with an interactive video stream. It has to be controllable mid-stream, responding to sound instructions as they come. And it has to be spatially aligned stereo, reflecting the geometry of the scene and the motion of the camera. Each of these pulls against the others. A system fast enough to keep up has little room to wait, a system that takes instructions has to change course partway through, and a system that places sound in space has to follow a camera that never stops moving.

WorldSonus is an interactive video-to-audio framework built for real-time spatial sound synthesis inside world models. The word interactive matters: the audio is not composed once for a finished clip, but produced alongside a stream that is still unfolding and can still be redirected. Its design answers the three demands one at a time, with a specific component for each.

For speed, it uses a streaming causal autoregressive diffusion architecture, producing audio in chunks at a low real-time factor of zero point four one. A factor below one means each chunk of sound takes less time to make than it takes to play. For control, it adds an audio-centric captioning pipeline with prompt scheduling indexed by chunk, so sound events can be manipulated dynamically while generation is under way. For spatial alignment, it trains on high-quality stereo supervision curated from diverse stereo and ambisonic data.

Although it is tailored to world models, extensive experiments show WorldSonus generalising to open-domain video-to-audio benchmarks. There it matches or outperforms state-of-the-art bidirectional models, both in acoustic quality and in spatial alignment.

That comparison is the telling one. A streaming, causal system gives up the ability to look ahead that a bidirectional model enjoys, and the reported result is that it holds its ground anyway, while also listening for instructions and placing sound in space.