HelixWorld: A Real-time Interactive Audio-Visual World Model

HelixWorld. A Real-time Interactive Audio-Visual World Model. By Lei Ke and fifteen others, from the Hong Kong University of Science and Technology and Noiz AI. Published on the twenty-ninth of September. Its code repository is worked out: about one hundred source files, with a readme and a licence.

Simulating a world is a multisensory job. What you see and what you hear have to stay synchronized, and they have to do it in real time, as the world responds to you. Yet interactive world models so far have stayed strictly silent. They focus entirely on rendering visuals and responding to control, and the acoustic dimension is simply left out.

HelixWorld is a real-time interactive audio-visual world model. Visual scenes and spatial stereo sound co-evolve natively as the user interacts, rather than sound being added on afterwards. The sound is grounded in the camera, so what you hear is tied to where the viewpoint is and how it moves.

The work starts with data: a curated, high-fidelity spatial audio-visual dataset carrying true stereo acoustics and metric camera poses. On it the authors pre-train a bidirectional teacher, conditioned on camera trajectories with six degrees of freedom and on user actions.

A bidirectional teacher looks in both directions, which does not suit low-latency causal interaction. So it is distilled into a few-step streaming student, using an online trajectory distillation loss. The student sustains joint audio-visual rollouts without drift, at twenty-four frames per second, on a single GPU.

To measure the new part of the problem, the authors formalize spatial-acoustic consistency and introduce HelixBench, which evaluates whether synthesized sound fields faithfully track the viewpoint as it moves.

In extensive experiments, HelixWorld matches state-of-the-art silent world models on visual fidelity and on responsiveness. On camera-aligned spatial-acoustic immersion, it significantly surpasses existing baselines.

The result to hold onto is that adding sound did not cost sight. The visual side keeps pace with models that only ever had to produce pictures, while the audio side follows the camera wherever the user takes it, all within real-time interaction on one GPU.