VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing

VidDub. Replacement-Aware Distillation for Two-Step Video Dubbing. By Yexin Liu and thirteen others, from the Hong Kong University of Science and Technology, Tencent, and four other institutions. Published on the second of November.

Video dubbing synthesizes lip movements that match a target speech track, while keeping the source video's speaker identity, head pose and background. Most existing editors learn this through self-reconstruction, trained on the original paired audio. The trouble shows up at the moment dubbing actually happens. Once the driving speech is replaced, the source mouth stops being the right answer. It becomes a conflicting target, pulling the edit back toward movements that belonged to the old audio. An editor trained only to reproduce the original frames has, in effect, been taught that the original mouth is correct, which is exactly what dubbing has to overrule.

The proposal is replacement-aware cross-audio supervision, which treats two kinds of sample differently. Paired-audio samples keep full-frame reconstruction. Unpaired samples, where the audio has been replaced, have their source-mouth reconstruction losses masked out. For those, the mouth is supervised only through audio-visual synchronization, while the unmasked facial regions and the background are still preserved. Everything outside the mouth is still held to the source. Only the mouth, the one region where the source and the new speech disagree, is released from it.

Offline dubbing offers one convenience: the complete source video and the target speech are both available in advance. VidDub takes advantage of that as a full-context diffusion editor, which is then distilled down to two denoising steps. Training proceeds in stages. A replacement-aware teacher first learns accurate lip synchronization without regressing toward the conflicting source mouth. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student. Last, distribution matching distillation recovers the visual quality lost to step compression. Each stage has a single job: first get the lips right, then make it fast, then restore how it looks.

Across three training seeds, masking out the conflicting source-mouth reconstruction raises cross-audio Sync-C by zero point eight three three. The final two-step model pairs strong lip synchronization with superior visual quality, identity preservation and visual speech recognition, across four benchmarks. The authors also say they will release the curated training dataset, VidDub-Bench, fixed evaluation manifests for the paired and cross-audio settings, and the preprocessing outputs.

The crux is what counts as a correct mouth. Under self-reconstruction it is the one in the source frame. Once the audio is swapped, that answer is wrong, and the method stops asking for it, letting synchronization with the new speech decide instead.