The Attention Triangle in Audio-Video Models

The Attention Triangle in Audio-Video Models. By Sagi Polaczek and six others, from Tel Aviv University and Simon Fraser University. Published on the third of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Audio-video diffusion models coordinate text, sound and visual content through cross-modal attention, and the claim here is that this same mechanism is where a subtle and systematic kind of semantic leakage comes from. The object of study is what the authors call the attention triangle: the three cross-attention edges connecting the text, audio and video streams. What is examined is how semantic information is routed across the modalities during generation, rather than the finished outputs alone. Probing those edges shows something particular about the audio-video one. Routing along it is bidirectional. Audio can influence video generation, and video can influence audio generation, along the same edge in both directions. That edge is shaped by biases encoded in the model's own parameters, and it emerges as a major contributor to leakage.

The mechanism is described concretely. When prompts are in tension with learned priors, cross-modal interactions may override the intended conditioning and reroute semantics toward outcomes that are visually canonical but incorrect. To study that, the authors extract attention-derived signals exposing how semantics are distributed and grounded across the modalities. Those signals serve as a diagnostic tool in two ways: they analyse leakage where it occurs, and they deliberately incur it under controlled conditions. Being able to cause the failure on purpose is what lets the internal dynamics of cross-modal routing be probed and the role of individual interactions isolated. The three edges are studied as a set, so what any single interaction contributes can be told apart from what the others do.

The claim is narrower and more specific than a general complaint about attention. The finding is not that attention spreads beyond its intended target, which would be a diffuse problem with no particular location. It is that semantic artifacts arise from structured, bias-driven interactions along specific pathways. That distinction is what makes the result actionable. A diffuse failure can only be traded against quality, whereas a failure living on a named edge can be intervened on. The same attention-derived signals are then used to guide inference-time interventions encouraging more consistent cross-modal alignment.

Extensive experiments support the analysis and demonstrate improved semantic grounding while preserving generation quality, and that pairing is the part that matters. An intervention which fixes grounding by degrading the output has fixed nothing. What the work leaves behind is less a technique than a way of looking: treat the cross-attention edges as a structure carrying its own biases, measure what flows along each one, and leakage stops being a mysterious artifact and becomes a routing problem with a location.

The Attention Triangle in Audio-Video Models. By Sagi Polaczek and six others, from Tel Aviv University and Simon Fraser University. Published on the third of September. This is the long version: the whole paper rather than the abstract.

The paper's opening figure repays a full description, because the whole argument is legible in it. The prompt asks for a weathered pirate in a black tricorn hat standing on a wooden dock at sunset, with a bright green and red macaw parrot perched on his shoulder. It then says something unusually explicit: the parrot is the only character that speaks, and the pirate does not move or speak. What the baseline model produces is a pirate who has inherited parrot-like attributes and who is doing the speaking. Two separate things went wrong. Appearance leaked from the parrot onto the pirate, and the sound was attributed to the wrong entity. The prompt said neither of those, and said the opposite of both.

Attention is why. Cross-attention is the control surface of modern diffusion generation, and it works by mixing information softly and globally, with no explicit locality or exclusivity constraints. That is a feature for compositional generation and a liability for binding. The image literature has the same failure in a simpler form: ask for a striped cat next to a spotted dog and you routinely get a striped dog, because patterns get routed by the semantics of the target entity rather than by the description attached to it. In audio-video generation the problem gains a dimension. Alignment is no longer pairwise but coupled across three modalities at once, and the model has to reconcile semantic intent from text, temporal structure from audio, and spatiotemporal realization in video.

The authors name that trimodal structure the attention triangle: three cross-attention edges connecting the text, audio and video streams, each modality both influencing and constraining the others. The first finding is that routing along the audio-video edge is bidirectional. Audio can influence the visual interpretation, and visual priors can bias the generation and attribution of sound, along the same edge. The second is that this edge behaves as a weak link. Where a prompt is in tension with the model's learned cross-modal biases, the audio-video interaction can override the binding the text established, rerouting sound semantics toward sources that are visually canonical and wrong. So leakage is not simply attention spreading past its target. It is bias-driven interaction travelling along one identifiable pathway.

The evidence that makes this more than a story is an ablation that decouples the two failures. Steering only the text edges restores the pirate's appearance, and leaves the speech still incorrectly localized to him. Steering only the audio-video edge does the reverse: it correctly localizes the speech to the parrot, and the appearance leakage persists. Only steering all edges of the triangle together resolves both. Two symptoms that looked like one bug turn out to live on different edges, and each can be fixed while the other is left standing.

The scoping is careful and stated in the discussion rather than buried. Their evidence establishes the contribution of the audio-video pathway; it does not establish a unique underlying mechanism. The spatial-versus-temporal token explanation is offered as a hypothesis, with the note that future work should directly test it. Everything is measured on one model. And an architectural question is left open that could limit the reach of all of it: whether the same leakage patterns and the same interventions transfer to single-stream models, which process every modality inside shared self-attention instead of through explicit cross-attention between modality-specific streams. If they do not, the triangle is a property of one family of designs.

The generalisation the authors draw is stated as a warning, and they name it the Attention Lesson: strengthening cross-modal coupling through attention can amplify bias-driven routing rather than resolve it. That runs against the natural intuition that more coupling means better alignment. Every new modality you wire into the attention graph is another route by which a learned prior can override what the prompt actually asked for, and the prompt in this case could hardly have been more explicit about which character was meant to speak.