Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning

Accent Analogy Guidance. More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning. By Yoomee Cho and one other, from ESTsoft. Published on the twenty-fourth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Cross-lingual zero-shot text-to-speech clones a voice from a reference recording and makes it speak another language. The trouble is that the reference carries its accent with it, and that accent leaks into the target speech. The obvious lever is to reweight classifier-free guidance between the reference and the text. Judged by a blind LLM accent judge on real dubbing data, that reweighting and its variants all stay near a single trade-off curve between identity and accent. Push toward a more native accent and you lose speaker similarity, and the reverse.

Accent analogy guidance, or AAG, is a training-free sampler term built to get off that curve. The analogy works like this. Take one synthetic voice and have the model render it in both languages. Because it is the same voice both times, the difference between the model's own predictions for the two renderings cancels the voice, and what remains is an accent direction. AAG subtracts that direction during sampling.

Measuring success needed its own framing. Since reweighting methods sit near one curve, a method is scored by how far its speaker similarity lies above that curve at an equal accent level. The paper calls this delta SIM. A method that merely slides along the trade-off scores nothing. One that buys identity without paying in accent scores above zero.

Across four open TTS models, AAG lies above the curve. On OmniVoice, delta SIM ranges from zero point one one to zero point two seven on three test sets. In those runs accent reaches between three point five one and four point two eight on a one to five scale, at a speaker similarity of zero point two nine, where reweighting keeps only zero point zero two.

MaskGCT and CosyVoice 2 also lie above their curves, and on F5-TTS, AAG is more native than any reweighting setting. Two checks that do not rely on the LLM judge agree: a language identification measure, and a panel of twelve listeners.

The most unusual result is a prediction. A premise test, together with how far a model's own curve reaches, indicates in advance whether AAG can help a given model and roughly by how much. That test singled out the one model where AAG gains nothing, X-Voice. A method that tells you beforehand where it will fail is easier to trust where it succeeds.