TaoMate-FVSR:Audio-Guided Facial Video Restoration
Xinyang Li, Zeran Xu, Qijun Gan, Chenwei Zhang, Meiguang Jin, Junfeng Ma
Alibaba Group
Published November 2, 2026 · on the feed November 3, 2026 · ▲ 2 · arXiv 2611.03393
The abstract
AI-generated videos often depict people within detailed scenes while leaving their small facial regions poorly resolved. Visual super-resolution can sharpen these regions without resolving whether the recovered lips and teeth agree with the spoken audio. We present an audio-guided facial video restoration framework that addresses this ambiguity within a reference-guided video flow model. Source--noise mixing initializes reconstruction from a partially preserved observation, and structured reference injection separates appearance anchors from continuous motion context. Training combines restoration of degraded images and videos with correction of degraded dubbed sources under target audio. Appearance and synchronization supervision allow the model to recover facial detail and support visual dubbing when the audio is replaced. On talking-head restoration benchmarks, our method achieves the best paired perceptual similarity and synchronization scores among the compared restoration methods. Component comparisons examine audio conditioning, the reconstruction formulation, and reference injection. Native AIGC examples further illustrate the recovery of lip contours, dental structure, and surrounding facial texture in scene-level footage. injection. Native AIGC examples further illustrate the recovery of lip contours, dental structure, and surrounding facial texture in scene-level footage.
The Brief
1:55A short podcast summary of the paper. Written and voiced by AI.
Transcript
TaoMate-FVSR. Audio-Guided Facial Video Restoration. By Xinyang Li and five others, from Alibaba Group. Published on the second of November.
AI-generated videos often show people inside detailed scenes, and their faces, small regions within the frame, are left poorly resolved. Visual super-resolution can sharpen those regions. What it cannot do on its own is settle whether the recovered lips and teeth actually match what is being said. A sharper mouth is not necessarily the right mouth for the audio.
TaoMate-FVSR brings the audio into the restoration. It is an audio-guided facial video restoration framework, built inside a reference-guided video flow model, and it uses the soundtrack to resolve that ambiguity about what the mouth should be doing.
Two design choices shape the reconstruction. Source and noise mixing starts the reconstruction from a partially preserved version of the observed video, rather than from scratch. Structured reference injection then keeps two kinds of guidance apart: appearance anchors on one side, continuous motion context on the other.
Training mixes two jobs. One is restoring degraded images and videos. The other is correcting degraded dubbed sources so they fit a target audio track. Supervision covers both appearance and synchronization. That combination lets the model recover facial detail, and also support visual dubbing when the audio is swapped for a different one.
On talking-head restoration benchmarks, the method reaches the best paired perceptual similarity and the best synchronization scores among the restoration methods it is compared with. Component comparisons look separately at the audio conditioning, the reconstruction formulation and the reference injection. Native AIGC examples, taken from scene-level footage, illustrate the recovery of lip contours, dental structure and the facial texture around them.
Restoring a face in a talking video is not only a question of sharpness. The mouth has to agree with the voice, and this framework treats the audio as evidence for what the missing detail should look like, which is also why the same model can redo the lips when the audio changes.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free