StepAudio 3 Gen Technical Report

StepAudio 3 Gen Technical Report. By Bin Lin and seventy others, from the StepFun Audio Team. Published on the eleventh of September. No code repository is linked to it.

Recent general audio models mostly generate continuously, following a diffusion Transformer paradigm. StepAudio 3 Gen sets out to cover a very wide range of audio in one unified framework, and departs from that paradigm to do it. The range includes zero-shot text-to-speech, voice design, vocal generation, sound effects, music, vibe speech, and mixtures of several audio types at once. That is a wide spread for one model, from spoken words in a designed voice, to singing, to music and effects, to several of these layered into the same clip. The challenge is a single representation, and a single generator, that can serve all of them.

At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization tokens. Its StepAudio Tokenizer represents general audio at twelve point five hertz, in a shared residual code space of sixteen by two thousand and forty-eight. It quantizes semantic and waveform-level acoustic features jointly, so that every code layer keeps both kinds of information. Meaning and sound are not split between different layers. Each layer of the code carries some of each.

Generation is split across two axes. The backbone predicts the first codebook along the time axis, autoregressively. A lightweight causal Transformer then completes the remaining fifteen codebooks along the codebook axis. The large model steps through time one frame at a time, and at each frame the small model fills in the remaining layers of the code for that frame.

The study identifies three design principles behind the system. Interference-aware progressive pretraining, which acquires audio capabilities while preserving the textual abilities of the large language model, so that learning to produce sound does not come at the cost of understanding text. An RVQ Adaptor, for incorporating multi-codebook acoustic representations effectively. And discrete autoregressive modeling over one shared representation across general audio domains.

Training proceeds through progressive pretraining, multi-task instruction training, and supervised fine-tuning. The result achieves state-of-the-art performance on both text-to-speech and voice design, while retaining strong generation across speech, vocals, sound effects and music. The state-of-the-art claim is made for text-to-speech and voice design specifically. For the other domains, the claim is that strong capability is kept alongside them. Audio samples are available on the project's page.

The distinctive choice is one discrete vocabulary for every kind of sound. Speech, singing, effects and music all become sequences of the same codes, and that shared representation is what lets a single autoregressive generator, built on a language model, handle them together.