SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

SepGen. Multi-Stem Audio-Video Separation and Generation in a Single Model. By Aviad Dahan and five others, from Tel Aviv University and Google. Published on the eighth of October. Its code repository is worked out: about forty-five source files, with a readme and a licence.

A 4D audio-visual scene has three parts. There is a video, there is the dynamic geometry the video depicts, and there are the sound sources that populate it, each with its own position and its own trajectory. Rendering such a scene from a novel viewpoint requires every source to be available as an individual waveform. Only then can each one be localized in the scene and propagated to the observer, before the signals are mixed together.

That requirement collides with how joint audio-video generators work today. They can synthesize a video and its soundtrack, but the soundtrack is emitted as a single audio-mix. The sources inside it are not individually accessible, so the one thing a novel-viewpoint rendering needs is exactly what the generator does not hand over.

SepGen extends a pretrained audio-video generator so that, in one joint sampling run, it emits three kinds of output: the video, the mixed soundtrack, and one waveform per captioned source. Those per-source waveforms are the stems. A single model and a single run produce the picture, the mix, and every source on its own.

SepGen supports two complementary modes. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker, in the original turn order.

In separation mode, the input audio-mix remains clean, and the captions instead specify what to extract. The same model therefore becomes a decomposer. It can pull sources out of an existing recording from a free-text description of what to find, rather than generating them from nothing. Between the two modes, the captions change role. In generation they describe what each stem should contain. In separation they name what should be pulled out of a mix that already exists.

Generation is evaluated on scenes synthesized from text. Separation is evaluated on scenes rendered by other generators, and on real recordings of speech, music and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech. It keeps the lead when the spoken lines are removed from the captions, leaving only the descriptions to go on. That second condition takes away the most direct hint about what each source is saying. The separation results, in other words, are not confined to material from SepGen itself: they include audio made by other generators and audio recorded in the real world.

The shift is from a soundtrack as one finished mix to a soundtrack as a set of addressable sources. That is what rendering a 4D scene from a new viewpoint calls for, and SepGen approaches it with one model that both creates stems from captions and recovers them from audio that already exists. Code, checkpoints and datasets are available.