Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

Prism. Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training. By Shuyuan Tu and ten others, from Fudan University, Tencent Hunyuan, and one other institution. Published on the fourth of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

Training a joint video-audio generation model natively at higher resolution lets it learn richer visual detail and sharper motion. Full attention stands in the way, for two separate reasons. The first is cost, which grows quadratically. The second is less obvious. As resolution climbs, full attention spreads itself over more and more redundant tokens. That dilutes the learning signal for the content that is actually informative, and it disrupts the priors the model learned in pretraining. So higher resolution makes full attention both more expensive and less useful.

Sparse attention is the natural response, but existing methods fall short here in one of two ways. Some target acceleration without training, which is a different job from training natively at high resolution. Others overlook the particular structure of joint video-audio data, where cross-modal interaction is concentrated around the regions that produce sound.

Prism is a dynamic sparse attention framework for training joint video-audio generation models natively at 2K. Its core move is to let the shape of the attention blocks follow the content, region by region, rather than imposing one partition everywhere.

First, Prism organizes the token sequence into spatiotemporal macro-zones, so the attention structure can adapt to local content. For each zone it estimates local information structure from two signals. One is the variance of video features along the channel, which captures how visual content varies directionally. The other is feature norms from the audio-to-video cross-attention, which capture how strongly audio influences each visual region.

From those signals, Prism assigns each zone a tailored block shape. Along axes where visual content changes quickly, or where audio-visual coupling is strong, the partition becomes finer. The intent is that tokens within a block stay semantically coherent, so block-level features can capture both visual content and the patterns of video-audio interaction. On top of that, a hybrid block selection strategy decides sparsity dynamically for each query.

Against full attention, Prism reaches a two point five times training speedup, and it surpasses full attention in generation quality as well. The abstract gives that single comparison, and it is the one the diagnosis calls for, since full attention was the thing said to be both costly and diluted.

Faster and better at once is the notable pairing. The problem statement explains it: at high resolution, full attention was not only expensive but also spread thin over redundant tokens. Concentrating attention around visual change and around the regions where sound and image are coupled is meant to answer both complaints with the same step.