Know, then Draw: A New Learning Paradigm for Visual Generation

Know, then Draw. A New Learning Paradigm for Visual Generation. By Yucheng Mao and four others, from UC San Diego and Adobe Research. Published on the twenty-seventh of October.

Generating an image asks for two different things at once. High-level semantic knowledge about what should be in the picture, and fine-grained pixel detail about how it should look. A model that knows what a scene contains can still render it poorly, and a model that renders crisp pixels can still get the content wrong. Whatever training recipe is used has to supply both kinds of ability.

The proposal is a learning paradigm that splits the two roles across two stages. Autoregressive pretraining is given the job of knowledge acquisition. Diffusion post-training is given the job of high-fidelity rendering. Know first, then draw. Each stage is asked to do the thing it is assigned, rather than one objective being asked to carry both.

Within that framework the authors study three questions: the objective, the representation, and the scaling of each stage. On the objective, controlled comparisons identify autoregression as the stronger pretraining choice for later rendering, improving both learning efficiency and final generation quality. On the representation, they find that generation quality from the autoregressive stage alone does not necessarily track final performance. Semantic quality matters more. That motivates a purely semantic tokenizer, trained with language supervision only, with no image decoder and no reconstruction loss.

On scaling, the authors establish compute-optimal scaling laws for knowledge acquisition and rendering. They find that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch. Guided by those laws, they train a text-to-image model with three billion parameters, using only six thousand six hundred H100 GPU-hours. It scores eighty-six on GenEval and eighty-seven on DPG-Bench.

Dividing the labour is the whole proposal: let one stage learn what things are and the other learn how they look. Two of the findings along the way reach beyond the recipe itself. The best pretraining for rendering is judged by semantic quality rather than by how well the first stage draws on its own, and a tokenizer can do its job without ever reconstructing an image. The scaling laws and the modest compute budget are offered as evidence that the division holds up at scale.