DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS

DEFINE. Exemplar-Guided Accent Control for Zero-Shot TTS. By Ambuj Mehrish and four others, from Ca' Foscari University of Venice, Kandinsky Lab, and two other institutions. Published on the thirtieth of September. Its code repository is public and real: about twenty source files.

Zero-shot text-to-speech can reproduce a speaker it has never heard, from a short reference recording. The difficulty is that the reference carries two things at once. Speaker identity and accent typically come entangled in the same clip, so the voice and the way it pronounces things arrive as a package.

DEFINE is an end-to-end framework that pulls the two apart, by conditioning speaker identity and target accent on separate audio exemplars: one recording for who is speaking, another for the accent. A single guidance weight, set at inference time, then controls accent strength continuously, without any retraining. Because the two references are separate, either one can change while the other stays fixed.

DEFINE is built on F5-TTS, with parameter-efficient LoRA adaptation. An exemplar encoder maps short accent exemplars into a conditioning space, and that encoder is supervised through learned accent prototypes. Two things are therefore not needed: accent labels at inference time, and any conversion of the waveform after synthesis. The accent, in other words, is taken from an example of how it sounds, organized around learned prototypes, rather than from a label someone has to supply, and the whole thing happens inside one model rather than in a second stage bolted on afterwards.

On seen accents, turning up accent guidance raises accent-probe accuracy from six point five percent to nineteen point six percent. More important is generalization. A single DEFINE model extends accent control beyond the set of accents it was trained on. On seen accents and on out-of-domain accents, it matches the accent transfer performance of a cascade pairing a TTS model with a voice-conversion model, while reaching higher speaker similarity and comparable predicted speech quality. On held-out accents, it does not match the cascade, and that exception marks the edge of the result.

The results support a separation of controls. Speaker identity and accent can be set independently from audio exemplars inside one zero-shot TTS model, including for accents unseen during training, and the strength of the accent becomes a dial rather than whatever the reference happened to contain. One model now does what took two in the cascade.