WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

WaveGen. Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning. By Sang-Hoon Lee and one other, from Ajou University and KT. Published on the ninth of November.

Recent text-to-speech pipelines lean on parts trained or decoded somewhere else. They pass through intermediate representations such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. They use external alignment modules such as duration predictors. And many depend on self-supervised representations, BERT-based text embeddings or Wav2Vec 2.0 semantic features, or on semantic distillation during training. Each of those is a component outside the model that the model relies on. A system built this way is not really one model learning to speak. It is several, trained separately and joined together, and the authors' phrase for what they want instead is a truly end-to-end formulation.

WaveGen sets out to remove all of them. It is a diffusion-based framework that goes from text straight to waveform, built on an approach the authors call Spectral Forcing. A single model jointly learns two trajectories: an internal spectral trajectory, and the trajectory of the target waveform itself. The spectral view is learned inside the model rather than supplied from outside, and text-speech alignment happens internally too, with no duration predictor. The same goes for training: no self-supervised representations and no semantic distillation are used to help optimisation along. Whatever the model needs to know about spectra, alignment or meaning, it has to work out within itself.

Spectral Forcing is realised as a model architecture with disentangled diffusion heads, trained on an objective based on a multi-scale spectral v-loss. Everything else is kept deliberately plain: one architecture, open data, and a single stage of training, with no external modules such as a duration predictor or REPA.

The authors also test whether end-to-end text-to-speech scales, growing the model from zero point one billion parameters to zero point six billion. The question there is whether a design stripped of every external helper still holds up as it grows, and the authors present the result as validating that it does. On the Emilia training dataset, WaveGen reports state-of-the-art performance for fully end-to-end waveform generation. On the Seed-en benchmark, it reaches a WER of one point four seven, and a SIM of zero point seven five.

WaveGen is defined as much by what it leaves out as by what it contains. No codec, no vocoder, no latent autoencoder, no aligner, no borrowed self-supervised features. What remains is one model that learns its own spectral picture on the way to the waveform, and the claim that this is enough.