Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion. By Kam Man Wu and six others, from the Hong Kong University of Science and Technology, Noiz AI, and one other institution. Published on the twenty-fifth of October. Its code repository is public and real: about thirty-five source files.

Authoring spatial audio means coordinating two kinds of decision at once. There are decisions about the sources: what sounds, and when. And there is the synthesis of the soundfield those sources live in. Editing adds a further demand. A plan has to say what to change and what to keep, while reference audio supplies acoustic detail that no plan spells out.

AMBIT is a unified framework for generating and editing native First-Order Ambisonics, or FOA, audio. At its centre is a shared Transfusion backbone that couples autoregressive planning with rectified-flow rendering. The link between the two is an executable ScenePlan, which specifies source content, source activity, and 3D trajectories. When editing, the plan describes the full target scene, retained sources included. Contrastive features ground the requested change, and full reference latents supply the residual acoustic detail that needs preserving.

Two further contributions sit around that backbone. The first is an FOA-aware continuous VAE. It combines shared-decoder omnidirectional recovery, an asymmetric grouped KL, and spatial covariance matching, with the aim of preserving both the content and the structure between channels.

The second is joint Transfusion on-policy self-distillation, called OPSD. It refines planning on prefixes the student generated itself, and refines rendering at the sampling states reached under the student's own plans. Target-conditioned teachers supervise both modes of the one shared student. In other words, the model is corrected in the situations it actually produces, for both the plan and the sound.

Evaluations cover speech, music, and sound. Against the compared baselines, AMBIT shows improved content and spatial quality in generation. In instruction-guided editing it shows better acoustic fidelity, and better preservation of the sources that were meant to stay unedited.

The design choice running through all three parts is that a plan is something to execute rather than a loose description. Generation and editing then become the same act: write the complete target scene, render it, and lean on the reference audio for whatever the plan cannot carry.