Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation

Hear the World in Stereo. Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation. By Hanmo Chen and nine others, from Xidian University and vivo BlueImage Lab at vivo Mobile Communication. Published on the eighth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Models that generate video and audio together have become strong at two kinds of agreement: semantic correspondence, and temporal synchronization. A third kind has been largely overlooked. Applications such as augmented and virtual reality, or interactive gaming, need stereo audio for a sense of immersion, and effective stereo means the place a sound seems to come from has to move consistently with the visual source producing it. The authors name this property Dynamic Spatial Correspondence.

StereoBind is a framework that binds the motion of a visual source to the stereo sound it generates. The common thread is motion tracks: they are what coordinates the visual motion with the stereo audio.

Three complementary mechanisms use those tracks. Visual Motion Binding establishes source-aware correspondence between sight and sound. The Spatial Track Encoder captures where each source is in absolute terms. Residual Track RoPE models how it moves relative to that. For supervision and evaluation, the authors build StereoWorld-29K, a large-scale stereo audio-video dataset paired with motion tracks, and StereoWorldBench, which measures audiovisual spatial consistency.

Experiments show that StereoBind substantially improves spatial alignment in stereo audio generation compared with existing models, while preserving overall audiovisual quality. The abstract states the result at that level, without figures. What it emphasises is the pairing: better spatial alignment, gained without giving up the quality that joint generation already had.

Reduced to one sentence, the proposal is that a moving sound should be generated from where its source is and how it is moving, with tracks of visual motion as the channel that carries that information from the picture into the stereo field. Each of the three mechanisms handles one part of that: which source a sound belongs to, where the source sits, and how it moves. The new dataset and benchmark exist because this property, Dynamic Spatial Correspondence, needed both supervision and a way to be measured before it could be improved.