Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes. By Wendi Sang and four others, from Qinghai University, Tsinghua University, and one other institution. Published on the twenty-third of October.

Audio-visual target speaker extraction pulls one person's voice out of a mix, using sight as well as sound. Done in streaming form, it is essential for applications where latency matters. Current causal systems, though, face two gaps that are coupled together. The first is in how they are tested. Standard benchmarks fix the number of speakers at two, and they ignore what real scenes contain: near-field and far-field interference, reverberation, and sudden noise. The second gap is cost. Causal architectures that perform well are too heavy to deploy at the edge. Lightweight alternatives try to win back capacity by invoking a small separator again and again, and that repetition eats the savings.

The paper answers each gap with its own contribution. For testing, it releases RealSSA, a realistic benchmark in two parts. For cost, it proposes Falcon, a causal extraction method whose Separator does its work in a single encoder-decoder pass rather than by repetition.

RealSSA-Sim provides controllable simulated scenes with between one and five dynamic near-field interfering speakers. Scenes with far-field interference go further and add between ten and fifteen far-field interfering speakers. RealSSA-Real is a held-out evaluation set recorded in the real world.

Inside Falcon, the Separator performs multi-scale time-frequency separation in that single pass. Training adds an encoder-space speaker contrastive loss. This loss suppresses leakage from near-field speakers, and because it acts only in training, it costs nothing at inference. It also transfers across backbones.

Falcon reaches state-of-the-art extraction quality on four benchmarks: RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. The efficiency gains are measured against the causal AV-TFGridNet baseline. Parameters drop by ninety-one point nine percent. Computation falls by a factor of twenty point two. GPU inference latency falls by a factor of eleven point one.

The two halves of the paper lean on each other. A lighter model is only convincing if it is tested in conditions that resemble where it will run, with many speakers, distance and noise. And a harder benchmark is most useful when something small enough for the edge can be measured on it. Falcon and RealSSA arrive together for that reason: one makes the scenes realistic, the other makes the extractor cheap enough to stream through them.

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes. By Wendi Sang and four others, from Qinghai University, Tsinghua University, and one other institution. Published on the twenty-third of October. This is the long version: the whole paper rather than the abstract.

Picture a hearing aid, or a video call, in a busy café. The job is to pull one person's voice out of the mixture, using the movement of their lips as the cue for whose voice to keep. This is audio-visual target speaker extraction, and for those uses it has to run causally: each output frame may depend only on what has already been heard and seen. The authors argue that two things hold the field back. The standard benchmarks mix exactly two speakers at controlled levels, with no reverberation control, no distant crowd and no sudden noise, and none of them come with real recordings to test on. And the causal models that do well are too heavy for edge hardware, while the light ones win back quality by running a small separator over and over, which spends the savings again.

The answer to the first problem is a benchmark called RealSSA. Its simulated half builds four kinds of scene: a small meeting room, a medium indoor space, a large public hall, and an outdoor street. Each mixture holds one to five nearby interfering talkers, and in the three larger scenes another ten to fifteen distant ones. The line between near and far is three metres from the microphone, chosen to approximate the distance beyond which reverberant energy outweighs the direct sound. Reverberation, ambient noise and sudden events like a horn or a slammed door are sampled together, so the difficulty changes along several axes at once. The second half, RealSSA-Real, is about eighty-four minutes recorded on a single phone, with ten participants across eight environments, never used for training.

The model is called Falcon. A frozen lip-reading network turns the mouth video into features, an audio encoder turns the waveform into a short-time spectrum, and the two are added together and handed to the Separator. The Separator is an asymmetric U-Net. Its encoder only shrinks the time-frequency map, halving both axes at each of three downsampling stages, and the heavy modelling happens at the coarsest level, which gives one compact summary of the whole scene. That summary is then injected back into every skip connection through a gate, which lets the global context emphasise or suppress frequency bands in the finer detail. On the way back up, each decoder level has its own refinement block: a frequency-wise recurrent layer, a time-wise one that only looks backward, then self-attention over the time-frequency plane with a mask that hides the future. Four blocks at four resolutions means four scales of modelling in one forward pass, where SwiftNet loops a single block many times.

The second idea lives only in training. Reconstruction loss pulls the estimate toward the clean target, but nothing pushes it away from a neighbour's voice. So the estimate, the clean target, and each near-field interferer are all passed through the same audio encoder, and pooled into unit-length speaker embeddings. The loss rewards similarity to the target, and penalises similarity to each interferer, but only while that similarity is above a margin, weighting the most similar interferers hardest as training goes on. Far-field talkers are left out, because their residual already sits below what reconstruction loss can see. At inference this branch is simply dropped, so it costs nothing.

On quality, the margins are small. Falcon has the best separation scores on all four simulated benchmarks, but on RealSSA's own simulated test it scores seven point seven decibels of improvement against seven point six for the audio-visual TF-GridNet, its strongest rival. The thesis lives in the cost. At the same parameter count as SwiftNet, Falcon needs seventy-six point six percent fewer operations, and its GPU time falls from one hundred and seventy-two milliseconds to about fifty-one. Against TF-GridNet it uses ninety-one point nine percent fewer parameters. The contrastive loss, tested on three different models, added between two tenths and four tenths of a decibel to each. And training on RealSSA rather than VoxCeleb2 lifted perceptual quality on the real recordings for all five architectures tested.

The paper has a short limitations paragraph naming three things. Everything is single-channel, though the simulation generates a four-microphone array. The real set is small: about eighty-four minutes and ten people. And the contrastive loss needs clean recordings of each interferer, which only simulation provides. The appendices add more. The latency figures are forward passes over a whole two-second clip, and the authors say true per-hop streaming latency still needs measuring with a streaming implementation. With all video lost, Falcon falls to minus nine point nine one decibels, so it depends on seeing the face. On the real recordings Falcon ranks second on word error rate and only ties SwiftNet on one perceptual score. No results report error bars or repeated runs.

The idea to carry away is that depth can stand in for repetition. Instead of running one small block many times, Falcon spends its budget once, across scales, in a single pass. Its depth is capped by the streaming budget too: the video imposes one hundred and twenty milliseconds of lookahead, which at an eight-millisecond hop allows fifteen audio frames, and a fourth stage would break that. Paired with a loss that teaches the encoder who the neighbours are, and that is then thrown away, this is a model shaped by deployment constraints.