YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

YuE2. Unifying Symbolic and Audio Music Generation at Frontier Quality. By Ruibin Yuan and thirty-four others, from the Hong Kong University of Science and Technology, H K G A I, and six other institutions. Published on the twenty-seventh of September. Its code repository is worked out: about fifty source files, with a readme and a licence.

Music generation has split into two camps. Symbolic models make the musical content explicit: melody, harmony, rhythm and form are all written down. But they typically stop before there is a finished recording. Audio models do produce complete songs, yet the composition inside them stays implicit, never written anywhere a person could read or change it. Each camp has what the other lacks. One gives you the sheet music but no performance, and the other gives you the performance with no sheet music behind it.

YuE2 joins the two through symbolic planning. One model first writes the music down as a readable score, and then performs it. The score is not a side output produced for show. It is the plan the audio is generated from, which means the composition becomes explicit without giving up a finished recording at the end. The claim is that this unifies symbolic and audio generation at frontier quality, rather than trading quality away for readability.

The model is a single AR-NAR Mixture-of-Transformers. It works in three moves. It writes a readable score that specifies melody and harmony. It expands that score into semantic music tokens. And it realizes those tokens as full-song audio. Each move hands a more concrete form to the next: first a score, then tokens, then sound. Training this pipeline from ordinary recordings raises a problem, because recordings do not come with aligned scores. The authors therefore introduce two supporting models. MERT2 supplies semantic supervision, and SheetSage2 supplies symbolic supervision.

The planning step is tested directly, using the same checkpoint with and without it, so that the only thing changing between the two conditions is whether a score gets written first. Experts prefer symbolic planning on overall quality and on musicality: it takes forty-nine point three percent of overall preferences, against thirty-four point six percent without planning. Experts also favour the unified model over a pipeline built from a separate language model and a diffusion Transformer.

On WildSongBench, YuE2 scores six point seven three on the SongBench Global Avg measure, ahead of every public baseline evaluated. Choosing the best of eight candidates lifts that to six point nine six, the highest mean observed among all the systems evaluated. In expert listening against proprietary song generators, the best-of-eight setting is preferred over Suno v4.5, and against Suno v5 preferences come out nearly balanced.

The supporting models hold up on their own terms. MERT2 sets a new state of the art in music representation learning, beating the previous best results on fourteen of fifteen MARBLE metrics. SheetSage2 leads on twelve of fifteen benchmark and metric pairs in the authors' comparison of lead-sheet transcription.

Because the score is readable, it becomes a handle. The same checkpoint follows edits made to the score while largely preserving the parts left untouched, and it produces covers zero-shot, without any training specific to covers. The score also opens the door to agentic editing, where external language models turn a user's feedback into revisions of the composition. Writing the music down first is what makes the finished song something a person can reach into and change.

YuE2. Unifying Symbolic and Audio Music Generation at Frontier Quality. By Ruibin Yuan and thirty-four others, from the Hong Kong University of Science and Technology, H K G A I, and six other institutions. Published on the twenty-seventh of September. This is the long version: the whole paper rather than the abstract.

A song generator usually hands you a finished recording and nothing else. The melody, the chords, the form, every decision that made the song what it is, stays buried in the audio. Symbolic music models have the opposite problem. They write a readable score, with melody, harmony, rhythm and form made explicit, but they stop before anything you could play as a finished song. YuE2 asks whether one model can do both: write the composition down first, in notation a musician can read, and then perform it as full-song stereo audio. The claim goes further than convenience. The paper argues that writing the score first makes the finished song better.

The framing is called progressive musical commitment. Text and lyrics state intent. A score commits to melody, harmony, rhythm and form. Performance and audio then settle timing, articulation, timbre, expression and production. YuE2 generates three things in that order. First, a score in ABC, a text-based music notation: tempo, meter and key in the headers, bar lines and section labels for form, chord symbols for harmony, and separate vocal and instrumental voices for melody. Second, semantic music tokens, twenty-five per second, a dense trajectory carrying what is awkward to write in a lead sheet. Third, continuous acoustic latents at the same rate, which a separately trained decoder turns into forty-eight kilohertz stereo.

The obstacle is data. Ordinary recordings do not come with aligned notes, chords, beats and sections. So every training target is built offline from the recordings themselves.

Two supporting models supply those targets. SheetSage2 is the transcriber. It reads a whole song through a full-context MERT2 encoder, frozen apart from low-rank adapters, and a six-layer autoregressive decoder writes one chronological event stream: beats, key, chords, sections and melody notes. A deterministic builder turns that stream into ABC. It was trained entirely on labels from an earlier non-autoregressive model, SheetSage2-Prober, and the student beats its teacher on ten of fifteen benchmark and metric pairs.

The semantic tokens come from a separate branch of MERT2, a music encoder with unusual training targets: frozen MuQ and Qwen2-Audio encoders are fused into one shared bottleneck and quantised, and decoders must reconstruct both teachers from the same codes. The tokenizer branch then switches to causal attention, learns to align with lyrics and reconstruct mel and chroma features, and finally gains a single codebook halfway up the network. One choice from it every forty milliseconds comes to three hundred and seventy-five bits per second.

The generator is one twenty-eight-layer backbone of about three point five eight billion parameters, built as a Mixture-of-Transformers. Each layer holds two experts side by side, one autoregressive and one not, with separate normalisation, projections and feed-forward weights but a single shared attention computation. The autoregressive stream predicts the score and then the semantic tokens, left to right, and cannot see the audio. The acoustic stream sees everything: the text, the full score, all the semantic tokens and every other audio frame. It predicts the latents by flow matching, updating all frames in parallel. Training covers about three hundred and forty-six thousand hours of music and mixes four tasks in balance: with both score and semantic tokens, without the score, without the tokens, and with neither. That mixture is what lets one checkpoint generate with or without a plan. Editing and covers use the same door: supply a revised score, or one transcribed from another recording, as a fixed prefix, and the model renders it.

The comparison that tests the thesis most directly is not against Suno. It is YuE2 against itself: same checkpoint, same prompts, same candidate budget, same decoder, with and without writing the score first. Experts preferred the planned songs for overall quality forty-nine point three percent of the time, against thirty-four point six percent for unplanned, with the rest ties. Melody and chord progression moved the same way. A second matched test sets the unified backbone against a separate language model and diffusion Transformer of similar size, both planning. Experts favoured the unified model on five of six criteria, with text alignment a near tie.

Does the audio actually follow the score? Transcribing each rendered song, melody and chord similarity to its own score are both above nine tenths, against roughly one fifth for a recording made from the other score for the same prompt.

Against proprietary systems the picture is mixed, and the paper reports it. Best-of-eight is preferred over Suno v4.5 and nearly tied with Suno v5. Suno v6 is preferred over it, fifty-nine point three percent against thirty-one point six, despite YuE2's higher automatic SongBench score. Audio quality is its clearest strength.

There is no limitations section, but the paper's own reports mark the edges. The cover experiment shows a trade-off rather than a clean win. With the full score, covers of unseen works stay recognisable, but as the chords and then the whole score are removed, musicality and style alignment rise while recognisability collapses. Holding the source harmony keeps the work's identity at the cost of the target style. The score-agreement figures rest on transcriptions that music experts checked and corrected. The expert panel was paid and recruited from conservatory and audio research backgrounds, and the melody and chord judgments came from just four listeners. One public baseline, HeartMuLa, used SongEval and AudioBox in its own post-training, which the tables flag.

The idea to carry away is that a readable intermediate need not cost quality. The same checkpoint, asked to write the score first, produced songs experts preferred, and the score then became the handle for everything else: local edits that leave the rest of the song intact, covers of works it never trained on, and a language-model agent that turned a Mandarin pop song, The Last Train, into English jazz by rewriting its score. The composition stays something you can read and revise before it becomes sound.