LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds

LiveFoley. Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds. By Yusheng Dai and eight others, from Monash University, Nanjing University, and four other institutions. Published on the fourth of November.

Interactive, immersive applications need sound generated from video as the video streams, not afterwards. Under real-time constraints, though, streaming video-to-audio methods fall far behind offline models, and the abstract locates two reasons. On the visual side, frontends shrink their receptive fields to cut look-ahead latency, and that badly degrades the alignment between what is seen and what is heard. On the audio side, causal autoregressive diffusion backends denoise each audio patch in isolation, which hurts audio quality. One bottleneck at each end.

LiveFoley is a streaming causal autoregressive diffusion architecture that takes on both bottlenecks, with latency the authors describe as imperceptible. Its answer has two halves: a fix for the visual frontend that keeps alignment without waiting long for future frames, and a fix for the audio backend that gives each denoising step more context.

For the frontend, a Trailing Sliding Window converts non-causal, segment-level encoders into streaming modules, with no retraining. Simply realigning the window preserves precise audio-visual alignment, and a configurable hop size pulls the frontend look-ahead well below the threshold of human perception. Because that hop size can be tuned, the authors build on it with the Unified Varhop Framework, which accommodates scenarios with different budgets for latency and computation.

For the backend, they explore widening the denoising window of the local diffusion transformer head, called LocDiT, in two temporal directions. History LocDiT adds past context. It improves quality at no cost in latency, and it is the default. The alternative, Future-Warmup LocDiT, breaks the autoregressive deadlock by supplying an approximate future condition. That improves generation quality further, at the cost of a backend look-ahead the authors describe as acceptable.

The experiments are described as extensive, and they report two things, one for each of the original bottlenecks. In temporal alignment, the place where streaming frontends had been losing ground, LiveFoley matches offline models, which do not have to work in real time at all. In generation quality, the place where isolated denoising had been hurting, it leads the streaming systems it is compared with.

Each half of the design answers one named bottleneck. The window keeps the pictures and the sound in step without a long look-ahead, and the widened denoising window lets each audio patch stop being generated alone. Around both sits a choice of operating points. The hop size sets how far ahead the frontend looks, and the backend can either add only past context, which is free in latency, or warm up on an approximate future, which costs a little look-ahead and buys quality. That is what lets one architecture be tuned to different latency and computation budgets, while matching offline models on alignment.