Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry
Szu-Chi Chen, Jia-Kai Dong, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee
National Taiwan University · NVIDIA
Published September 27, 2026 · on the feed September 29, 2026 · ▲ 6 · arXiv 2609.33999
The abstract
Speaker verification (SV) models are commonly assumed to better capture nuances among speaker characteristics as verification accuracy improves, leading to their widespread use as automated proxies for human voice similarity in speech generation tasks. However, by establishing a human perceptual alignment metric and conducting systematic analysis, we demonstrate that perceptual alignment is governed far more by how a model is trained (its learning objective) than by how well it performs (EER). Notably, standard margin-based classification losses (e.g., AAM-Softmax) yield substantially lower perceptual alignment than prototypical metric losses, while EER itself fails to track human judgment, directly challenging the community's implicit assumption. We trace this divergence to embedding geometry, where a model's effective dimensionality (d_{eff}) tracks perceptual alignment with a -0.95 rank correlation, revealing that the dimensional spread favored by classification losses fundamentally clashes with the low-dimensional nature of human voice perception. Imposing a dimensionality bottleneck compresses d_{eff} and raises perceptual alignment (ρ_{align}) from 0.08 to 0.74, establishing a principled geometric criterion for evaluating voice similarity.
The Brief
2:09A short podcast summary of the paper. Written and voiced by AI.
Transcript
Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry. By Szu-Chi Chen and four others, from National Taiwan University and NVIDIA. Published on the twenty-seventh of September. No code repository is linked to it.
Speech generation research routinely uses speaker verification models as automatic stand-ins for human judgments of how similar two voices sound. The assumption behind that habit is that a model which verifies speakers more accurately must also capture the nuances of a voice better. This paper disputes it. How well a model agrees with human perception, the authors find, depends far more on how the model was trained, meaning its learning objective, than on how well it performs as measured by equal error rate, or EER.
The authors first establish a metric for alignment with human perception, then analyse models systematically. Standard margin-based classification losses, AAM-Softmax among them, produce substantially lower perceptual alignment than prototypical metric losses do. EER, meanwhile, fails to track human judgment. A model can sit near the top on verification accuracy and still be a poor judge of which voices people hear as alike.
The divergence traces back to the geometry of the embeddings. A model's effective dimensionality tracks its perceptual alignment with a rank correlation of minus zero point nine five. The reading offered is that classification losses favour spreading embeddings across many dimensions, which clashes with the low-dimensional nature of how humans perceive voices. The test that follows is direct: impose a dimensionality bottleneck, and effective dimensionality shrinks while perceptual alignment rises from zero point zero eight to zero point seven four.
The claim concerns speaker verification models used as proxies for voice similarity, and it is measured against the authors' own human perceptual alignment metric. What it overturns is a specific assumption, that better verification means better similarity judgment, not the usefulness of these models for verification itself.
For anyone scoring generated speech with a verification model, the practical upshot is to check the geometry rather than the leaderboard. A compact embedding space, the authors argue, is a principled criterion for judging voice similarity, and a lower error rate is not.
Paper Quest · 7 chapters · about 37 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a habit nobody questioned. Here it is trusting the best verifier to hear voices like we do.
You build speech generators, and every paper you read scores voice cloning the same way: run both voices through a speaker verification model and take the cosine similarity of their embeddings.
The road ahead
- 1 · The problem ~7 min Speech papers score cloned voices with a speaker verification model, trusting that the most accurate verifier also hears similarity like people do.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min How a speaker model's embedding space is shaped by its training objective, not its verification error, decides whether it judges voice similarity like listeners.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~7 min You’ll guess: What best predicted how well a speaker verification model's judgments matched human perception of voice similarity?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.