Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation

Nemotron-Labs-Diffusion-Uni. Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation. By Shufan Li and seven others, from NVIDIA. Published on the twenty-seventh of October.

Nemotron-Labs-Diffusion-Uni, shortened to NLD-Uni, is a unified discrete diffusion large language model. One model is meant to handle both visual understanding and visual generation. The abstract positions it against prior unified diffusion models, and names two places where it departs from them: where its abilities come from, and how its tokens are corrupted during training.

The first is a unified cross-paradigm on-policy distillation framework. It transfers skills from models built on other paradigms. Visual understanding comes from autoregressive VLMs. Visual generation comes from continuous flow models. Both are carried into a single unified diffusion language model, so the student learns from teachers that work quite differently from itself.

The second concerns corruption. The conventional masked diffusion formulation corrupts a clean image token directly into an uninformative mask token, in one move. NLD-Uni instead uses a progressive self-refining diffusion process, which corrupts each token gradually across multiple steps during training. At inference, that design lets the model make an intermediate commitment to a token and keep refining it, which the authors credit with better visual quality.

In extensive experiments, NLD-Uni reaches state-of-the-art performance among unified diffusion language models. The tasks are diverse: visual question answering, multimodal reasoning, object grounding, text-to-image generation, and image editing. That list spans both halves of the model's remit, understanding on one side and generation on the other.

One model, two borrowed skill sets, and a gradual corruption process that lets a token be committed early and still refined. Those are the pieces the authors offer for bringing seeing and drawing together inside one diffusion model.

Nemotron-Labs-Diffusion-Uni. Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation. By Shufan Li and seven others, from NVIDIA. Published on the twenty-seventh of October. This is the long version: the whole paper rather than the abstract.

A unified diffusion language model reads and writes both text and images with one objective. It fills in masked tokens in parallel, which makes it faster than an autoregressive model that writes one token at a time. The problem is quality. These models still trail the specialists: autoregressive vision language models on understanding, and continuous flow models on image generation. The obvious fix is to learn from those specialists, and that is exactly where things break. On-policy distillation, where a teacher grades the student's own outputs, normally needs teacher and student to generate the same way. No strong masked diffusion teacher exists. The strong teachers write left to right, or they work in a continuous latent space. The paper's answer is Nemotron-Labs-Diffusion-Uni, built to learn across that divide.

Two pieces of background matter. The first is the language backbone. It generates in blocks: blocks come one after another, but tokens inside a block are unmasked in parallel over several steps. Shrink the block to a single token and the model becomes an ordinary autoregressive model. One network, then, carries two views of itself: an autoregressive view and a diffusion view.

The second is what masked diffusion does to images. Each image patch is a discrete code from a vector quantizer. At inference every code starts masked and, at some step, jumps straight to a final value and never changes again. Codes unmasked at the same step are sampled independently, as if unrelated, though neighbouring patches in an image are strongly correlated. A continuous flow model can keep revising. A masked one cannot.

The fix for understanding is transitive. The student rolls out answers in its autoregressive view, at a low temperature. The frozen teacher supervises that view with the standard on-policy loss. Then the clean block is partially masked, and the student's diffusion view is trained to match the student's own autoregressive prediction, with the gradient stopped on that target and the loss computed only at masked positions. The teacher never speaks to the diffusion view directly. Its knowledge passes through the student's autoregressive view, whose context the student's own rollout fits.

Generation needs a different trick, because no exact distribution matching is possible against a model in continuous latent space. The student edits an image. That output is re-encoded with the teacher's continuous autoencoder, noised, and re-denoised by Qwen-Image over several steps, conditioned on the instruction and source image. The result, which the authors call a teacher-refined student output, stays close to what the student produced but with fewer artifacts, and becomes an ordinary supervised target. The authors admit this is not on-policy in the conventional sense, but argue the target follows the student's evolving outputs.

The second idea gives image tokens an age. Age zero is clean, the top age is fully masked, and in between are intermediate states. Corruption follows a balanced binary ball tree over the codebook's embedding space: each step up the tree replaces the token with a random code from a larger neighbourhood, until at the root it could be anything. A token is degraded coarse to fine, not erased in one jump. The tree is chosen for efficiency: a full pairwise distance matrix would take thirty-two gigabytes, while the eighteen-level tree fits in a single array of five hundred and twenty-four kilobytes. Each token also gets its own time window during which it passes through these ages, so at inference a token can commit to an intermediate guess and keep refining it.

The comparison that tests the distillation thesis is the naive baseline. On multimodal reasoning, the pre-distillation checkpoint scores seventy-one point five on MathVista. Copying teacher scores onto the diffusion view drops it to forty-seven point seven, worse than doing nothing. The cross-paradigm method lifts it to seventy-seven, with the largest gain on We-Math, about fifteen points. Image editing tells the same story: the base model's ImgEdit score of three point eight one rises to four point one seven, while training on independent teacher images reaches only three point nine. Only GPT-4o and the Qwen-Image teacher itself score higher.

For self-refinement, the appendix holds the tuning that matters. At sixty-four steps, eight refinements per token works best. At very few steps, refinement actively hurts, and the single-jump schedule wins. On the visual-quality score HPSv3, the model reaches eleven point one two, above both Flux.1-dev and Qwen-Image. Speed holds up too: about five hundred and fifty tokens per second against about two hundred and three for its own autoregressive mode, at a cost of about one point in MathVista accuracy.

The paper has a limitations section, and it is specific. Distillation still requires teacher and student to share a vocabulary; here both use the Ministral 3 tokenizer, and cross-tokenizer distillation is left to future work. The self-refining process uses fixed corruption windows and uniform progression, chosen to keep complexity down rather than because it is optimal. The model cannot render fine text, because such data is scarce in its training mix. Beyond what is stated: the reported runs do not show error bars or repeated seeds, the teacher-choice study used an earlier checkpoint, and the authors credit most of the gains on general understanding benchmarks to scaling training data rather than to either new method. Training took sixty-four H100 GPUs for twenty-six days.

The idea to carry away is that a student can learn from a teacher that does not speak its language, provided the lesson is translated into the student's own terms. For text, the translation is the student's autoregressive view. For images, it is the teacher lightly correcting the student's own picture rather than painting a new one. In both cases what failed was a teacher target pulled far from the student's own outputs, and what worked was supervision anchored to them.