Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy

Joint and Cross-Modal Video-Audio Generation and Editing. A Unified Formulation and Design Taxonomy. By Abhinav Sharma and twenty-three others, from the University of Massachusetts Amherst, the University of Texas at Dallas, and eleven other institutions. Published on the twenty-eighth of September. No code repository is linked to it.

People see and hear at the same time, yet most generative models handle video and audio in isolation. This overview gathers the methods that do not. Some model the two modalities jointly, some generate one from the other, and some edit them in a coupled way. One question runs through all of it: how is the output kept coherent across the two modalities, both in time and in meaning?

The organising device is a unified formulation. Joint generation, cross-modal generation and joint editing are cast as three problems defined on a single distribution over audio-visual pairs. On top of that sits a taxonomy that compares methods along five design axes. Joint audio-visual editing gets its own map, nine edit categories spanning twenty-eight edit types, which the authors believe is the first systematic taxonomy of that area.

For each of the three settings the overview describes the methods, the datasets and the metrics. The coherence question is the thread that ties them together, since the same pair of modalities has to stay aligned whether it is being generated jointly, translated across, or edited.

The overview closes on the open problems the authors view as most consequential. What a listener carries away is the framing: three tasks that are usually handled apart, recast as one distribution over audio and video seen from three angles, with editing given a map of its own.