Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation

Intern Lumina U2. A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation. By Qian Yu and forty-two others, from the Shanghai AI Laboratory, the Shanghai Innovation Institute, and nine other institutions. Published on the fourteenth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.

Unified multimodal generation comes in three paradigms: discrete, continuous and hybrid. Each pays a price. Discrete approaches that use a visual tokenizer with a single codebook line up naturally with text modeling, but their representational capacity is limited. Continuous and hybrid approaches keep richer visual representations, but they model text and images in different spaces with different training objectives. That can introduce optimization conflicts between modalities whose representation spaces do not line up. So the trade looks stark. Stay discrete and share one space with text, but accept thinner visual tokens. Or keep the richer visuals, and accept that text and images are being optimized as two different kinds of thing.

The central question this exposes is how to model high-capacity visual representations alongside text, in one shared optimization space, inside a single backbone, and to do it efficiently.

Intern Lumina U2 answers with a diffusion language model that represents visual tokens from several codebooks in a fully discrete space. To the authors' knowledge, it is the first to do so. Each visual position is represented not by one code but by a group of codes, drawn from complementary codebooks. That raises the capacity of each position, while everything stays discrete and so remains compatible with discrete text modeling. Think of a single word versus a short phrase in the same spot: the phrase carries more, yet it is still made of the same kind of discrete symbols.

At the input, the embeddings of a position's codes are concatenated. At the output, a lightweight autoregressive head predicts the codes within each position one after another, while the diffusion backbone processes all positions in parallel. Sequential inside a position, parallel across them. The two kinds of order sit at two different levels, and each is handled by a different part of the model: the small head for the codes that share a position, the diffusion backbone for the image as a whole.

The model is built on the unified 4D representation of AToken, which lets it take image, video and 3D inputs within the one backbone. The authors also build a full-stack system for training, inference and evaluation on both Ascend and NVIDIA platforms, to verify that the model works across platforms.

Among unified models, Intern Lumina U2 achieves leading performance in visual understanding, in image generation, and in image editing. On speed, a stability-aware sampling strategy improves inference efficiency. It combines an unmasking approach guided by confidence plateaus with a mechanism that refreshes the cache when drift is detected, and together these accelerate image generation by four point four five times. Leading results are claimed on three fronts at once, understanding, generation and editing, from a single backbone that also accepts video and 3D inputs, and the cross-platform system is there to show the model is not tied to one kind of hardware.

The design choice at the centre is to buy visual capacity by widening each position rather than leaving the discrete space. Several codebooks per position give the richness that single-codebook tokenizers lacked, and keeping every code discrete lets text and images share one optimization space and one backbone, which is exactly the tension the problem set out.