Text-to-Image Generative Modeling Without Language Pretraining

Text-to-Image Generative Modeling Without Language Pretraining. By Keshigeyan Chandrasegaran and seven others, from Stanford University and Radical Numerics. Published on the nineteenth of October. No code repository is linked to it.

Modern text-to-image systems use internet-scale language pretraining to condition image generation on text. The authors ask whether that pretraining is necessary for strong text-to-image generation, and their answer is that it is not. Strong systems of this kind can be trained from scratch without it.

The evidence is Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of one hundred million image-text pairs. Halcyon-0.52B outperforms LlamaGen-XL/8, at a gFD of forty-eight against sixty-three point three, with eighteen point two times less pretraining compute.

At larger size, Halcyon-1.6B outperforms JiT-T2I-XXL/8++, at thirty-five point three against forty-one point zero gFD, with two hundred and seventy-five times less pretraining compute. Controlled experiments then test the question directly: text-only pretraining does not improve text-to-image generation.

Halcyon also learns image captioning, by replacing caption dropout with image-to-text supervision, and it keeps competitive text-to-image quality under the same pretraining budget.

The comparisons are made against two named systems, using gFD. And what is shown to be unnecessary is internet-scale language pretraining, not text itself: Halcyon still learns from a hundred million image-text pairs.

On this evidence, language pretraining moves from prerequisite to option for text-to-image generation. The language such a model needs can be learned from the paired captions it trains on, at a small fraction of the compute.