Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing
Rui Liu, Bhavin Jawade, Haoqi Li, Shivam Mehta, Karan Saxena, Yinghong Lan, Cameron R. Wolfe
University of Maryland, College Park · Netflix
Published September 30, 2026 · on the feed October 2, 2026 · ▲ 17 · arXiv 2610.00825
The abstract
Dubbing quality control requires a reference-free judge that can determine whether a candidate text line matches a speaker's visible articulation in both content and timing, using only silent video and text because dubbed audio may not yet exist. Existing visual speech recognizers and video-language models are poorly suited to this setting: even when fine-tuned to recover spoken content from lip motion, they remain largely insensitive to temporal errors. We introduce Align Then Reason (ATR), a multilingual lip-sync judge that first establishes a monotonic alignment between frame-level lip representations and the phonetic units of the candidate line, then reasons over this alignment to make the final judgment. An alignment scorer provides the LLM with both local evidence for each phonetic unit and a calibrated global alignment score, enabling it to reason jointly about content and timing. On a seven-language benchmark, our method improves mean AUC over the corresponding Qwen3.5 SFT baselines by 59.4%, 50.2%, and 50.8% with 2B, 4B, and 9B reasoners, respectively. The gains generalize across LLM families, reaching mean AUC improvements of 45.9% and 46.6% over the best baseline for LLaMA-3.1-8B and Mistral-7B, respectively. They also transfer across datasets to three unseen MuAViC languages. Furthermore, we evaluate on two downstream tasks built from real dubbing lines. On dub-line reranking, ATR-9B outperforms the best lip-reading baseline by 52.0%, while on script-to-clip assignment, ATR-9B improves over the best lip-reading baseline by 17.7%.
The Brief
2:33A short podcast summary of the paper. Written and voiced by AI.
Transcript
Align Then Reason. A Multimodal Lip-Sync Judge for Dubbing. By Rui Liu and six others, from the University of Maryland, College Park and Netflix. Published on the thirtieth of September. No code repository is linked to it.
Quality control for dubbing needs a judge that works without a reference. Given a candidate line of text, it has to decide whether that line matches what a speaker's lips visibly do. Two things have to match: the content, and the timing. And the judge has to work from silent video and text alone, because at that stage the dubbed audio may not exist yet. Existing visual speech recognizers and video-language models are poorly suited to the job. Even when they are fine-tuned to recover spoken content from lip motion, they remain largely insensitive to errors in timing. They can tell you roughly what was said, but not reliably whether it was said at the right moment.
Align Then Reason, or ATR, is a multilingual lip-sync judge that does what its name says, in that order. First it establishes a monotonic alignment between frame-level lip representations and the phonetic units of the candidate line. Monotonic here means the alignment only moves forward through the video, as speech does. Then, with that alignment in hand, it reasons over it to reach a final judgment.
The two halves meet through an alignment scorer. It hands the language model two kinds of evidence. One is local evidence for each phonetic unit. The other is a calibrated global alignment score for the line as a whole. Holding both, the model can reason jointly about content and about timing, rather than about content alone.
On a benchmark spanning seven languages, ATR improves mean AUC over the corresponding Qwen3.5 SFT baselines by fifty-nine point four percent with a two billion parameter reasoner. With four billion it improves by fifty point two percent, and with nine billion by fifty point eight percent. The gains generalize across language model families. Over the best baseline, mean AUC improves by forty-five point nine percent for LLaMA-3.1-8B, and by forty-six point six percent for Mistral-7B.
They also transfer across datasets, to three unseen MuAViC languages. Two downstream tasks are built from real dubbing lines. On reranking dub lines, ATR-9B outperforms the best lip-reading baseline by fifty-two percent. On assigning scripts to clips, it improves over the best lip-reading baseline by seventeen point seven percent.
The design is an ordering, and the title states it. Settle where each sound falls against the lips first, and only then ask a language model for a verdict, so that the reasoner is handed timing as evidence right alongside content.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The voice doesn't exist yet. Only silent lips and a line of text. Does it fit?
You review dubbed lines for films. Before any voice is recorded, you must decide whether a translated line fits what the actor's lips visibly do, in both words and timing, from silent video and text.
The road ahead
- 1 · The problem ~7 min A dubbing judge must tell, from silent video and text alone, whether a line fits the lips in both words and timing.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Match lip frames to the line's sound units along a forward-only path, then let a language model reason over local and global match evidence.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Encode lips and sounds, align them in order, hand local and global evidence to a language model, and let it judge.
- 6 · The experiment ~6 min You’ll guess: With a 9B reasoner on the seven-language benchmark, how much did ATR improve mean AUC over the matching Qwen3.5 SFT baseline?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.