NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech
Qiaolin Wang, Pedro Sandoval-Segura, Anunaya Joshi, Edvardas Jurkonis, Jake Downie
Bland AI
Published September 25, 2026 · on the feed September 29, 2026 · ▲ 3 · arXiv 2609.31892
The abstract
While modern text-to-speech (TTS) systems generate highly natural speech and support inline non-verbal vocalization (NVV) tags, accurate control over these events remains challenging. A key gap is the lack of established post-training methods for non-verbal control in continuous autoregressive flow-matching TTS. To this end, we present NVAlign, a direct-gradient post-training framework for NVV tag-following in this architecture. We first perform supervised fine-tuning (SFT) of TTS models and an NVV-aware automatic speech recognition (NV-ASR) model on NVV-annotated speech, then freeze the NV-ASR model to serve as the reward model for post-training. A two-step gradient surrogate enables efficient reward backpropagation through the flow-matching sampler to jointly update the autoregressive backbone and acoustic flow head. Fidelity penalties and reference-velocity regularization help preserve speaker similarity and speech quality. Results from NVV-SuperBench and human listening evaluations show that NVAlign improves tag-following accuracy over SFT and Flow-GRPO baselines. These findings demonstrate that direct reward-gradient optimization can improve non-verbal control in continuous autoregressive flow-matching TTS. Audio samples are available at https://nvalign.github.io/.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
NVAlign. Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech. By Qiaolin Wang and four others, from Bland A I. Published on the twenty-fifth of September. No code repository is linked to it.
Modern text-to-speech systems sound highly natural, and many let you place tags inline in the text for non-verbal vocalizations, the sounds of speech that are not words. Getting the model to actually produce those events where the tags say is still hard. For one family of systems in particular, the authors see a clear gap. Continuous autoregressive flow-matching TTS has no established post-training method for non-verbal control.
NVAlign fills that gap with a direct-gradient post-training framework for following non-verbal tags, designed for this architecture specifically. The reward comes from a speech recognizer trained to notice those vocalizations, and the gradient of that reward is pushed straight back into the speech generator. The model is not only told whether a tagged sound came out right. It is shown, through the gradient, which way to move to make it come out right, and the whole generator is adjusted accordingly.
Training runs in two parts. First comes supervised fine-tuning, both of the TTS models and of an NVV-aware speech recognition model, called NV-ASR, all on speech annotated with non-verbal vocalizations. NV-ASR is then frozen and serves as the reward model. To get the reward's gradient through the flow-matching sampler efficiently, NVAlign uses a two-step gradient surrogate, which lets it update the autoregressive backbone and the acoustic flow head together. Fidelity penalties and a reference-velocity regularization are added to help keep speaker similarity and speech quality intact while the tag-following improves.
The evaluation uses NVV-SuperBench along with human listening tests. On both, NVAlign improves tag-following accuracy over two baselines: plain supervised fine-tuning, and Flow-GRPO. The first shows what post-training adds beyond fine-tuning alone. The second is a comparison against a different route to the same goal. The abstract does not give the margins in numbers.
The authors read the result as evidence for a general point, not only for their system. In continuous autoregressive flow-matching TTS, optimizing directly on the gradient of a reward can improve non-verbal control, with a frozen recognizer standing in as the judge of whether each vocalization arrived where its tag asked for it.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Natural speech is solved. Laughing on cue is not.
Your text-to-speech model sounds natural, and the text can carry inline tags like a laugh or a breath. But the model often ignores them or puts the sound in the wrong place.
The road ahead
- 1 · The problem ~7 min Speech models can read tags like a laugh or a breath in the text, but they often skip the sound or put it in the wrong place.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min A frozen tag-aware speech recogniser scores each output, and its gradient is pushed straight back into the speech generator, with guards on voice and quality.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Fine-tune the speech model and a tag-aware recogniser, freeze the recogniser as judge, then backpropagate its reward through a two-step shortcut into backbone and flow head.
- 6 · The experiment ~6 min You’ll guess: On following non-verbal vocalization tags, how did NVAlign compare with the SFT and Flow-GRPO baselines?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.