All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
Ohad Rahamim, Dvir Samuel, Idan Schwartz, Gal Chechik
Bar-Ilan University · NVIDIA
Published September 23, 2026 · on the feed September 24, 2026 · ▲ 8 · arXiv 2609.27901
The abstract
Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event. We find that joint multimodal diffusion transformers exhibit a corresponding asymmetry in cross-modal correspondence: companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. We express both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap. We introduce RecCAR, standing for Reciprocal Cross-modal Attention Regularization, a KL regularizer that uses the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it. Across joint video-motion and video-audio generation, RecCAR improves the Human Anatomy score from 0.69 to 0.75 and reduces audio-video desynchronization from 0.804 to 0.752, while improving overall generation
The Brief
2:02A short podcast summary of the paper. Written and voiced by AI.
Transcript
All modalities are equal, but video is more equal. Closing the Cross-Attention Gap in Joint Video Generation. By Ohad Rahamim and three others, from Bar-Ilan University and NVIDIA. Published on the twenty-third of September. Its code repository is a placeholder: a readme, and nothing to run yet.
Video is a rich representation of a physical event: appearance, geometry, motion, and how all of these evolve over time. Other modalities, such as three-D body motion or audio, encode narrower aspects of the same event. The authors find a corresponding asymmetry inside joint multimodal diffusion transformers. Companion modalities develop strong correspondences to video, but the reciprocal correspondences, through which they would constrain the video, remain substantially weaker.
RecCAR, short for Reciprocal Cross-modal Attention Regularization, targets that imbalance. It uses the strong direction as the reference for the weak one. Rather than inventing a target for the weak direction, it borrows one that the model has already learned well.
First, both directions are expressed as comparable correspondence distributions over video tokens, and their disagreement is defined as the reciprocal correspondence gap. RecCAR is then a KL regularizer. It takes the well- established video-to-modality correspondence as a fixed reference, and aligns the weaker modality-to-video correspondence toward it.
RecCAR is tested on joint video-motion generation and joint video-audio generation. It improves the Human Anatomy score from zero point six nine to zero point seven five, and reduces audio-video desynchronization from zero point eight zero four to zero point seven five two, while also improving overall generation. One result comes from each setting: anatomy for motion, synchronization for audio.
The title's borrowed line carries the finding: all modalities are equal, but video is more equal. Correspondence runs strongly one way and weakly the other, and RecCAR closes part of that gap by using the direction that already works as the template for the one that does not.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Two streams, wired both ways. Only one of them is listening.
You use a model that generates video together with 3D body motion, or with audio, in one process. Each stream can attend to the other, in both directions.
The road ahead
- 1 · The problem ~7 min A model that makes video and a companion stream together can link them both ways, yet one direction of that link barely gets used.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Use the model's own strong video-to-companion attention as the target, and pull the weak reverse direction towards it with a KL loss.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Put both attention directions over video tokens, measure their gap with KL, then train the weak direction towards the fixed strong one.
- 6 · The experiment ~6 min You’ll guess: Applied to joint video-motion generation, what did RecCAR do to the Human Anatomy score?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.