Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Youtu-Parsing-Omni. One Encoder, One Schema, All Modalities. By Yunhang Shen and twenty-nine others. Published on the twenty-ninth of October.

Agents that read, locate, and cite evidence need a perception layer underneath them, one that can take documents, images, audio, and video and turn them into structured records. Those records need to be grounded in space and in time, so that evidence can be pointed to, whatever modality it came from. A table on a page, a spoken sentence, and a moment in a video are very different inputs, and the challenge is to bring them into one form that an agent can locate and cite in the same way.

Youtu-Parsing-Omni is an omni-modal parser built to be that layer. Its design rests on two unifications, and the subtitle names them. One encoder for every modality. One schema for every task.

The encoder is Youtu-Omni-Encoder. It is initialized from a pre-trained language model, to strengthen text reading, and it encodes all modalities with shared weights in a unified space-time coordinate system. A few fusion layers let video frames attend to the audio playing at the same moment, which brings cross-modal fusion and temporal alignment into perception itself rather than leaving them for later.

The schema is OmniSchema, which unifies the outputs of all ten parsing tasks. Because the outputs share one form, a single self-evolving post-training procedure can serve every task. It is called OmniSchema-Aware On-Policy Distillation, or OSAD, and it provides a shared objective based on token roles. In each round the model is trained on its own rollouts, against token-level targets produced by a reference-conditioned copy of itself. That copy is then refreshed with the trained weights. The parser and its supervision evolve together: as the model improves, so does the copy that produces its targets for the next round.

All of the results come from a single checkpoint of five billion parameters. It achieves state-of-the-art performance on OmniDocBench v1.6, at ninety-six point nine six Overall. On OmniParsingBench it leads all open-weight models with an average of seventy-five point zero eight, and its advantage there is concentrated in audio and video.

It also achieves the best overall chart-parsing score, ninety-five point zero two, and the highest exact agreement on chemical fingerprints among the systems compared. On typeset sheet music, it matches a dedicated optical-music model in character-level transcription, while staying far ahead of general-purpose systems.

The economy here is in how few parts there are. One set of encoder weights reads pages, pictures, sound, and moving images. One output schema covers ten tasks. And because the schema is shared, one training loop can improve all of them at once, with the model's own refreshed copy acting as its teacher in each round.