A Language-Informed Loss for Visual Generation

A Language-Informed Loss for Visual Generation. By Suchir Agarwal and six others, from Stanford University. Published on the sixteenth of October. No code repository is linked to it.

End-to-end visual generation trained with a loss in pixel space has recently reached competitive performance, without needing a learned image tokenizer. The authors ask whether supervision informed by language could give visual generation a richer training signal than pixel loss does.

Their answer is LILIE, pronounced lily: a Language-Informed and Lossless Image Encoder. It is an invertible image transform, trained so that a vision-language model can recover captions from its features after those features have been noised. An encoder trained under that demand has to keep what a caption depends on recoverable, even through noise.

The encoder is a normalizing flow transformer, trained jointly with a vision-language model through next token prediction. Once trained, it is used to extract a simple language-informed loss for text-to-image generation. Because the transform is invertible, it is lossless: moving into its feature space throws none of the image away.

The authors train JiT models with the LILIE loss. On the GPIC dataset, against a pixel-only baseline, gFD falls by fourteen points: from sixty-six point eight four to fifty-three point zero one. The code, the encoders and the generative models are all released.

The loss is the bridge between the two halves. A model that generates pixels is graded in a space shaped by what language can recover from an image, and that space is still a complete description of the image. The fourteen point drop in gFD is the authors' evidence that such a signal is richer than pixels alone.