Unlocking Any-Order Generation in Pretrained Autoregressive Image Models

Unlocking Any-Order Generation in Pretrained Autoregressive Image Models. By Rishav Pramanik and two others, from Stony Brook University, the International Laboratory on Learning Systems, and one other institution. Published on the twenty-ninth of October.

Autoregressive image generators are powerful in principle and inflexible in practice. Trained in a fixed raster-scan order, a model has one particular factorization of the image distribution hardcoded into it. At inference time it cannot inpaint, cannot outpaint, and cannot edit. Existing remedies, the modern discrete-diffusion family among them, train from scratch on a mixture of token orderings. That costs roughly three times the compute of standard raster-scan training. Discrete diffusion also gives up, by construction, compatibility with the autoregressive inference ecosystem, things like KV caching and vLLM. So the choice on offer has been an unhappy one. Keep the raster-scan model and its inference tooling, and give up editing. Or gain editing by paying for a new training run several times over, and in the discrete-diffusion case, lose the tooling as well.

The authors argue that this cost is unnecessary. A pretrained raster-scan model has already learned the joint distribution over image tokens. What it lacks is not knowledge. It lacks the ability to express that knowledge through conditional pathways that are not sequential. So rather than training a new model, the paper offers a recipe for adapting a pretrained raster-scan model into an any-order generator.

The recipe has two components. The first is future-aware positional embeddings. These resolve what the authors call a target-position identifiability problem, and do so at negligible overhead. The second is an ordering curriculum, which traces a stable optimization path from the pretrained solution to a good any-order basin. Through all of it, the standard autoregressive inference stack is left untouched. One piece addresses what the model can be told about where it is generating. The other addresses how to get there from the pretrained weights without losing the way, moving gradually from the order the model knows toward the many orders it is meant to handle.

The recipe is applied to four base models, spanning two architectures and two scales: LlamaGen-L and XL, and RAR-L and XL. On ImageNet inpainting it substantially outperforms MaskGIT, with an FID of six point two one against ten point four four, and it also outperforms MaskGIT on outpainting. It is competitive with MAR-L.

The flexibility has a price, and the abstract states it. Class-conditional FID rises by between one point two and one point nine. The adaptation costs between thirteen and thirty-three percent additional compute beyond pretraining, set against the roughly threefold cost of training on mixed orderings from scratch.

The argument rests on a distinction between knowing and expressing. If the raster-scan model already holds the joint distribution, then inpainting, outpainting, and editing do not have to be built in from the start. As the authors put it, they need not be added to autoregressive image models. They can be unlocked from the ones that already exist, at a fraction of the cost and without leaving the inference ecosystem those models were built for.