AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

AuK Technical Report. An Open-Source Foundational Model for Speech Generation and Editing. By Ziyang Ma and thirty-two others. Published on the eighth of September. Its code repository is public and real: about thirty source files.

Speech generation and speech editing are usually separate capabilities. AuK is an open-source foundational model that unifies them behind one common interface: instructions in natural language, together with audio context. Covering that broad a set of abilities takes a great deal of supervision, and a model able to generate, edit, enhance and restore speech has to be trained on all of it.

The data comes first. The authors construct approximately three point zero three billion instruction and audio instances, and one point nine five million hours of effective supervision. They span five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. The range runs from producing speech outright, to changing what is said, to cleaning up or separating audio, to adjusting how something is said, to altering its acoustic surroundings. All five are expressed through the same instruction interface.

The model has three parts. A multimodal large language model supplies semantic conditioning. A VAE, trained jointly on speech, general audio and music, supplies acoustic conditioning. Training that VAE on music and general audio as well as speech gives the acoustic side a broader base than speech alone. And a hybrid rectified-flow Transformer generates, first through dual-stream MMDiT blocks and then through unified single-stream DiT blocks. Training starts with a warm-up on generation alone, then moves to joint pre-training on generation and editing. Post-training uses two complementary strategies: preference optimisation from human feedback for open-ended editing, and reward-based reinforcement learning for speech generation. Each kind of task gets the kind of feedback suited to it: human preferences where an edit is open-ended, and rewards where generation can be scored.

To cut inference cost, the model is distilled further, with consistency initialisation and a task-routed Decoupled DMD. The result is AuK-Flash. It runs inference in four steps, without classifier-free guidance, and achieves a four point five times wall-clock speedup over the full model under matched conditions. The comparison is with the full model, run under the same conditions.

Experiments show leading performance on zero-shot speech generation, on instruction-controlled speech generation, and on general instruction-guided editing. On signal-level restoration tasks the model remains competitive. Both the source code and the model weights are released.

The shape of the work is breadth first, then speed. One instruction-driven model covers five families of audio tasks, trained in stages from generation to joint editing to targeted post-training, and a distilled version makes it much faster to run.