UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

UniAVC. Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling. By Kien T. Pham and eight others, from the Hong Kong University of Science and Technology and Huawei Research. Published on the seventeenth of October. Its code repository is worked out: about sixty source files, with a readme and a licence.

A unified audio-video creation model has to do a lot from one set of weights. It has to generate sound and picture together from text, generate either one from the other, and edit both following instructions. The paper identifies two obstacles.

The first is data. Existing methods lean heavily on paired audio-video corpora, which are scarce for joint generation and scarcer still for editing. So that precious paired data ends up spent teaching each branch to generate and edit on its own, something plentiful single-modality data could teach easily, instead of teaching how the two modalities couple.

The second is the noise schedule. Prevailing systems use one shared schedule for both modalities, and that misses two things. Video and audio differ a lot in information density. Video needs a long stretch at high noise to organise its spatial and motion structure, while the sparser audio signal can settle earlier, and a uniform schedule may compromise those dynamics. And denoising both in lockstep leaves asymmetric workflows untrained, such as conditioning on one modality to generate or edit the other.

UniAVC answers both. For data, it trains in stages, so that the cheap, abundant kind of data does the work it is able to do, and the scarce paired kind is saved for the one thing only it can teach. For scheduling, it lets each modality keep its own clock, rather than forcing sound and picture through the same denoising trajectory at the same pace.

Training first brings each branch, on unimodal data, to full competence at generation and editing. Only then is paired data brought in, and it is dedicated to learning the correspondence between the modalities.

Meanwhile, modality-decoupled schedulers warp the noise schedule separately for each modality, starting from a shared timestep. Alongside that, subtasks in which one modality is kept clean directly supervise the asymmetric configurations that lockstep denoising had left untrained. Holding one side clean while the other is denoised is, in effect, a direct rehearsal of the asymmetric workflows the paper wants the model to handle: generate one modality given the other, or edit one while the other stays put.

Those two settings unlock two capabilities. One is intent-aware editing, which keeps one modality intact while using it to constrain the other. The other is self-refinement after inference: if one branch comes out unsatisfactory, the model regenerates only that branch, against the one that was accepted. The abstract reports these as capabilities rather than giving numbers, and its summary claim is that UniAVC spans multiple creation tasks from a single set of weights, with no task-specific adapters.

The unifying thought is that audio and video do not need to be treated as one thing moving in step. Let each branch learn what it can alone, let each settle at its own pace, and spend the scarce paired data only on what truly requires both.