StepAudio 3 Music Technical Report

StepAudio 3 Music Technical Report. By Chengli Feng and twelve others, from StepFun, ACE, and two other institutions. Published on the eleventh of September. No code repository is linked to it.

StepAudio 3 Music is a large-scale model for long-form music generation, with two stated aims: explicit musical planning, and open-domain generation controlled by text. The planning aim is the distinctive one. Most of the design that follows is about giving the model a representation of music it can plan over, and then turning that plan into high-fidelity audio.

The design is discrete and continuous at once. The StepAudio Music Tokenizer represents audio as a fifty-hertz stream drawn from a single codebook of sixty-five thousand five hundred and thirty-six entries. It is trained with semantically informed self-supervised and multi-task objectives, so the tokens keep both musical structure and the information needed for reconstruction. A flow-matching diffusion Transformer, or DiT, then predicts continuous StepAudio VAE latents, and the VAE decoder turns those latents into forty-eight kilohertz audio. Discrete tokens carry the structure, and the continuous stage carries the sound, so neither has to do both jobs. That split was guided by comparing single-codebook VQ, semantic and acoustic RVQ, and different DiT configurations.

Planning happens before any music tokens are produced. A Mixture-of-Experts autoregressive model first writes an intermediate arrangement plan in ABC notation, called ABC-CoT. That makes harmony, rhythm and melodic structure part of the generation context, rather than something left to emerge.

A progressive training curriculum and supervised fine-tuning then support several tasks: song and instrumental generation, accompaniment generated from dry vocals, and cover-song synthesis, for pieces up to five minutes and thirty seconds long. Reinforcement learning via direct preference optimization, or DPO, comes last. So the pipeline runs from symbolic plan, to discrete tokens, to continuous latents, to audio, with each stage handing a more concrete description to the next.

Among the systems evaluated, the final model has the highest AudioBox scores for Content Enjoyment, Content Usefulness and Production Quality, along with the highest MuQ-MuLan similarity, and competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, its Quality Elo is eleven hundred and five. That places it behind only Suno V5.5 and Mureka, and ahead of Suno V5, MiniMax models, and other systems.

The notable step is writing the arrangement down before playing it. A symbolic plan in ABC notation turns the structure of a song into something the model conditions on, rather than something it has to discover sound by sound. Audio demonstrations are available on the project page.