VoxLock: Target-Speaker Extraction for Voice Agents Without an Enrollment Step

VoxLock. Target-Speaker Extraction for Voice Agents Without an Enrollment Step. By Arbaz Siddiqui, working independently. Published on the twenty-ninth of October.

A voice agent's speech recognizer writes down everyone near the microphone, not just the caller. Someone talking in the background ends up in the transcript as if the caller had said it.

VoxLock keeps the caller and drops the rest, without asking the caller to enroll a voice sample first. The author took a public causal target-speaker extraction model, REAL-TSE, at twenty-seven million parameters, and fine-tuned it on a single GPU to work from a voice sample cut out of the call itself.

Evaluation is judged by the output that matters to the agent: what a recognizer actually writes on real recordings. The released model is an eight-bit version sized for live calls.

On Krisp's benchmark of two hundred and sixty-five recordings, VoxLock reaches a word error rate of four point nine five percent, against twenty-three point two one percent with no processing. Over eighty-five minutes of background-only speech, it lets nine background words into the transcript. Its error rate is not significantly different from Krisp's best filters, and is significantly lower than the other commercial and open systems tested.

A second test uses four hundred AMI meeting windows, built and frozen before any scoring. There, the word error rate falls from forty-two point nine five percent to ten point six three percent. In use, the eight-bit model runs a live call on seventy-seven percent of one CPU thread, with twenty-four milliseconds of algorithmic latency.

Three numbers carry it: an error rate brought under five percent from over twenty-three on real recordings, nine stray words in well over an hour of other people talking, and less than one CPU thread to run it, all without an enrollment step.

VoxLock. Target-Speaker Extraction for Voice Agents Without an Enrollment Step. By Arbaz Siddiqui, working independently. Published on the twenty-ninth of October. This is the long version: the whole paper rather than the abstract.

A voice agent hears whoever is near the microphone: a coworker at the next desk, a television, a second person in the car. Their words land in the transcript, and the agent answers them. Noise suppressors do not help, because they are built to keep speech, and all of this is speech. Krisp, ai-coustics and Sanas sell voice isolation filters for exactly this problem, but they are closed and licensed per minute or per device. VoxLock is one independent researcher's open alternative, built on a single GPU.

The research name for the problem is target-speaker extraction. Given a mixture and a short enrollment recording of one speaker, output only that speaker. Two things keep it out of voice agents. An agent has no enrollment recording, only the call itself. And these systems are usually scored with signal metrics on simulated mixtures, while an agent only cares what the recognizer writes on real audio. The starting point here is a public causal model from the REAL-TSE challenge baseline: a band-split recurrent network of twenty-seven million parameters with twenty-four milliseconds of algorithmic latency. Three metrics carry the evaluation. Word error rate on the caller's words. Missed words, the share of the caller's speech the transcript loses. And leak: words the recognizer writes where only background voices are present, per minute of such audio.

The first idea is where the voice sample comes from. The model takes exactly three seconds of the caller, cut from the call. For a whole recording, it uses the loudest three seconds within the first fifteen. For a live stream, audio passes through untouched until three seconds of speech have been heard, counting short frames that sit well above the quietest recent frame; those frames become the sample, and isolation switches on. Both rules assume the caller is the loudest, earliest talker, which holds for a headset or a phone held by the caller. The sample then enters the network twice. Each frame of the call's spectrum attends over the frames of the sample's spectrum, giving an extra input channel. And the separator's band features attend over speaker features of the sample from an ECAPA-TDNN encoder, with the result multiplied into every band.

Training runs in four stages, each starting from the last, all minimising negative scale-invariant signal-to-distortion ratio. Stage one simulates calls: a near-field LibriSpeech caller mixed with one or two quieter background talkers, some in simulated rooms, a quarter passed through a telephone channel. Stage two is the clever one. An AMI meeting headset records its wearer loudly and everyone else quietly. When the wearer talks alone, that is an exact clean target. When only others talk, that is real crosstalk. Mixing the two gives real training pairs without any separated ground truth. Stages three and four exist because of failures the author saw. Trained on six second windows, the model let a caller who talked without pause fade by twenty decibels within forty seconds, so stage three trains on thirty second examples. And the model had learned to keep whichever talker was closer, so stage four places the caller near, mid or far independently of the background.

On Krisp's benchmark of two hundred and sixty-five real recordings, VoxLock reaches four point nine five percent word error rate, against twenty-three point two one unprocessed. It is not significantly different from Krisp's Default filter, and significantly better than Krisp's Lite model, both ai-coustics models, Sanas, and every open system tested. Leak is starker: over eighty-five point three minutes of background-only speech, nine background words reached the VoxLock transcript, against a hundred and thirty for Krisp Default. There is a cost when nothing needs removing. In plain phone calls, VoxLock raises word error rate from three point eight two to five point three six percent.

The most direct test is the blind set: four hundred windows from AMI test meetings, built after the model was final and frozen by hash before scoring. There, word error rate falls from forty-two point nine five percent to ten point six three, and the untouched starting checkpoint sits at fourteen point seven five. Then comes a sharper probe: feed the wrong person's voice sample. The starting checkpoint follows that person and loses sixty-one percent of the caller's words. VoxLock loses seventeen percent, against eight with the right sample. The stage table adds one more thing the abstract does not: most of the gain comes from simulated stage one, and the first thousand steps of real crosstalk were no better than a thousand more simulated steps.

The paper has a limitations section, and it is candid. The model is English-trained, sixteen kilohertz and single-caller. The automatic sample assumes the caller is the loudest early talker. Both comparison sets were seen during development, and every AMI window starts with the caller, which flatters a sample taken from the start. The Krisp benchmark comes from a vendor being compared against, under a non-commercial license. All transcripts come from one recognizer family, Parakeet, and all results from one training run. The intervals are uncorrected for multiple comparisons, and the stage table is not a controlled ablation, since each stage was designed after seeing the last. And the blind set has no commercial rivals on it, so the head-to-head comparison rests on sets the author had seen.

The idea to carry away is that the enrollment step was never essential. For a voice agent, the caller is usually the loudest, earliest voice, and three seconds cut from the call itself is enough to steer a small public model. The result is statistically level with Krisp's best filters on their own benchmark, and the eight-bit model runs a live call on seventy-seven percent of one CPU thread.