SEAM: Spectral Embedding Alignment and Merging for Low-Resource TTS

SEAM. Spectral Embedding Alignment and Merging for Low-Resource TTS. By Walaa Othman and three others, from ITMO University, the St. Petersburg Federal Research Center of the Russian Academy of Sciences, and one other institution. Published on the nineteenth of October. No code repository is linked to it.

Adding new words to the vocabulary of a conditional text-to-speech model typically means extensive fine-tuning with paired audio. For low-resource languages, and for cross-lingual adaptation, that paired audio is exactly what is scarce, so vocabulary expansion becomes a major bottleneck. The model cannot learn to speak a new token without audio of that token, and the audio is the thing nobody has.

SEAM is a zero-audio initialization method. It builds embeddings for tokens the model has never seen out of subtokens the model already knows, before any audio is involved. The new entry starts from pieces the model has already learned, rather than from nothing.

Two steps set it apart. Standard pooling strategies combine subtoken embeddings while ignoring their sequential structure. SEAM instead applies a spectral filter, which preserves the temporal order of the phonetic content. It then uses closed-form ridge regression to align the new embeddings with the pretrained conditioning manifold, so they land where the model expects conditioning to live.

The method is tested across three languages and two TTS architectures. There, SEAM approaches or matches baseline performance using up to four times less training data. Against random initialization and mean pooling, that amounts to a seventy-five percent reduction in sample complexity. Put another way, a quarter of the data reaches roughly where the usual starting points need all of it.

The authors present it as lightweight, reproducible and theoretically grounded. In practice that comes down to starting new tokens in the right place, so that far less audio is needed to finish the job.

SEAM. Spectral Embedding Alignment and Merging for Low-Resource TTS. By Walaa Othman and three others, from ITMO University, the St. Petersburg Federal Research Center of the Russian Academy of Sciences, and one other institution. Published on the nineteenth of October. This is the long version: the whole paper rather than the abstract.

A text-to-speech model trained on English knows a fixed vocabulary of tokens, and each token has an embedding that the generator conditions on. Point it at Russian or Kazakh and new tokens appear that it has never seen. Each one needs an embedding before fine-tuning can start, and in practice those embeddings are drawn at random and then trained with plenty of paired audio. For a low-resource language, that audio is precisely what is missing. The common shortcut, borrowed from language modelling, splits the new token into known subtokens and averages their embeddings. The paper's complaint about averaging is specific: an average does not care about order. The same pieces in any arrangement give the same vector, and speech is nothing if not ordered.

Two ideas set up the method. The first is a continuity assumption. A well-trained generator is taken to be smooth in its conditioning: nearby embeddings produce nearby audio, which the appendix states formally as a Lipschitz bound on the expected output. The consequence is that a new embedding should land close to where the pretrained ones live, because an erratic vector produces erratic sound and unstable gradients. The second idea is to treat a token's subtokens as a short signal. Stack the subtoken embeddings in order, and each embedding dimension becomes a sequence over positions, something a Fourier transform can be applied to. Low frequencies carry the slow, overall shape of that sequence. High frequencies, the authors argue, carry the sharp jumps at the boundaries between subtokens, which they treat as artifacts to suppress.

SEAM, short for Spectral Embedding Alignment and Merging, works in two stages. The first is the merging operator. For each embedding dimension, it takes the discrete Fourier transform across subtoken positions and multiplies it by a Gaussian low-pass filter. It keeps at most four frequency bins, and fewer for very short tokens, since a short sequence only has about half its length plus one unique bins. The real and imaginary parts of the kept coefficients are flattened into one feature vector. Keeping the imaginary parts is the point: they hold phase, and phase encodes which subtoken came first. An appendix proves that reversal changes the output under stated conditions, and admits those conditions may fail for very short decompositions.

The second stage turns that feature vector into an embedding the generator understands. A linear map does it, and the trick is where its weights come from. The pretrained vocabulary already contains many tokens that its own tokenizer can split into smaller pretrained tokens. Each one is a free training pair: the spectral features of its pieces as input, its real embedding as target. Stack those pairs and solve a ridge regression in closed form. No gradient descent, and no target-language audio. A new Russian or Kazakh word is first transliterated into an English phonetic representation, split into existing subtokens, passed through the filter and the learned map, and rescaled to the average norm of the pretrained embeddings.

The main backbone is Chatterbox, fully fine-tuned on Russian, Kazakh and French at budgets of five, twenty-five and one hundred hours. Intelligibility is scored by a speech recognizer as character error rate, averaged over three seeds. At twenty-five hours on Russian, SEAM reaches eight point five percent against ten point seven two for averaging, already better than averaging trained on four times the data. Kazakh shows the widest gap: SEAM stays near five percent at every budget, while averaging never gets below eight.

The comparison that most directly tests the thesis is the order ablation. Keep the whole pipeline, but reverse or shuffle the subtokens before filtering. On Russian at five hours, character error rises from fourteen point eight eight to nineteen point six seven when reversed, and to twenty-two point nine one when shuffled, which is worse than plain averaging. Same filter, same regression, only the order changed, so the gain comes from order rather than from the alignment step alone. Another appendix ablation is the surprise: refining SEAM's embeddings on one hour of real audio before fine-tuning made things worse. Embedding norms inflated, the loss spiked to fifteen, and it settled near ten, against roughly four for zero-audio SEAM.

The paper does have a limitations section, with three entries. Initialization depends on reliable subtoken decomposition and transliteration, and mapping errors can propagate. Fine-tuning with paired audio is still required, just less of it. And evaluation at tiny budgets is noisy, as in Kazakh, where random and average initialization got worse between five and twenty-five hours.

The evidence has further edges that the abstract smooths over. The four times data saving holds for Russian and Kazakh, but not French: there SEAM at twenty-five hours scores eleven point nine three, while averaging at one hundred hours reaches nine point zero six. At one hundred hours, averaging edges SEAM on French, and random edges it on Russian. The second architecture, VibeVoice, rests on a single Russian run at five hours, as do the extended baselines and the order ablation. Three seeds are averaged, but no error bars are reported. In the human study, SEAM at twenty-five hours was preferred over averaging at one hundred hours in every language, but the French bars, forty-eight percent against thirty-two, do not give the seventy point six percent non-tie share that the statistics table reports.

The idea to carry away is that initialization is a place to put structure the model would otherwise have to buy with data. Averaging throws order away. SEAM keeps it in the phase of a few Fourier coefficients, and borrows the pretrained vocabulary's own decompositions to learn, in one linear solve, how such features map to embeddings. The payoff shows up where data is scarcest, and on Russian and French it largely fades by one hundred hours: an initialization that gets the model started, then gets out of the way.