EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

EmoRES-TTS. Residual-Enhanced Vector Steering for Emotional Speech Generation. By Kuan-Po Huang and seven others, from Reality Labs at Meta, FAIR at Meta, and one other institution. Published on the twenty-ninth of September. Its code repository is public and real: eleven source files.

Ask an emotion-conditioned text-to-speech model for a particular emotion, and it may not reliably express it. Fixing that with more training is costly, both in computation and in speech data labelled with emotions. Vector steering is a training-free alternative: it modifies the internal representations of a frozen model instead of updating its weights. The conventional steering method for emotional speech, CoCoEmo, treats each emotion vector as a single indivisible direction, turned up or down by one global strength. That limits how closely the output adheres to the emotion requested.

The key observation is that an emotion vector is not indivisible after all. It can be decomposed into two components. A shared component moves speech away from a neutral expression in general. A residual component points generation toward the specific emotion requested. Moving away from neutral and arriving at the right emotion are two different things, and one dial cannot set both.

Emotion Residual-Enhanced Steering, EmoRES for short, controls those two components separately, and it does so without retraining the backbone. The model stays frozen; only the steering changes. That keeps the method inside the same training-free budget as the vector steering it improves on: no extra computation spent on training, and no new emotion-labelled speech to collect. Where CoCoEmo had one strength to set for a whole emotion vector, EmoRES treats the shared part and the residual part as two things to be handled on their own terms. The abstract does not spell out the exact settings for each; what it reports is that the two are controlled separately.

Tests run on IEMOCAP, with two backbones, IndexTTS-2 and CosyVoice2. On both, EmoRES beats CoCoEmo on all four objective emotion metrics. Rank correlation rises by twenty-six point one three percentage points on one backbone and by twelve point nine seven on the other. Those are relative gains of one hundred and eighteen point eight percent, and thirty-three point one percent.

Emotion hit rate rises by twelve point nine five points on one backbone and by six point nine two on the other, relative gains of twenty point one percent and nine point eight percent.

Objective metrics are one thing; the authors also put the speech in front of people. The rate at which they correctly identified the dominant requested emotion improves by up to thirty-five percent in relative terms. Fidelity improves by up to seventeen point three percent. And in pairwise comparisons for naturalness, listeners prefer EmoRES in up to sixty-three point eight percent of cases.

Component ablations settle which part does the work. Effective control benefits from leaving the shared component intact while strengthening the residual. The move away from neutral should be left as it is; the part to amplify is the push toward the particular emotion, which a single global strength could not single out.