FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

FLAT. Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation. By Guangyu Sun and ten others, from Meta AI. Published on the fifteenth of September. No code repository is linked to it.

Traditional multimodal representation learning and generation happen in two stages. A contrastive or self-supervised visual encoder is trained first, and a separate downstream generative model is built on top of it. That arrangement bottlenecks generative performance behind frozen embeddings. The encoder was trained to describe, and the generator has to make do with whatever it was given.

FLAT, for Flexible-Length Aligned Transmodal representations, revisits joint training of the two. It is a representation pre-training framework in which a shared multimodal encoder is optimized together with downstream decoders for text-to-image, or T2I, and image-to-text, or I2T. Contrastive alignment is combined with bidirectional cross-modal generative objectives, so the representations have to work both as discriminative semantic descriptors and as conditions for generation. The target is embeddings that interpolate linearly, and that generative decoders can consume directly.

Architecturally, visual and textual inputs are both mapped into one continuous one-dimensional sequence space. Nested dropout is applied over the first K tokens, which enables dynamic output lengths: a representation can be cut to a shorter prefix and still be usable. After a single pre-training stage, FLAT performs cross-modal retrieval and generation across a variable prefix K. One model, trained once, can therefore be asked for a short representation or a long one, and the same representation can be used either to search or to generate.

From pre-training alone, FLAT scores seventy-one point one on GenEval for text-to-image. Task-specific fine-tuning brings it in line with state-of-the-art baselines. On GenEval, that is eighty-three point one. On MS-COCO image captioning, it is forty point five BLEU-4 and one hundred and thirty-eight point six CIDEr.

For retrieval on MS-COCO, Recall at five is eighty-six point eight from image to text, and seventy-five point eight from text to image. On Flickr30K, the same measure is ninety-eight point three from image to text, and ninety-three point six in the other direction. Qualitative evaluations show the representations natively support linear interpolation, latent space arithmetic, and zero-shot composed retrieval.

The bet is that one representation can serve both search and synthesis, as long as generation is part of how it is learned rather than something bolted on afterwards. The variable-length prefix adds a dial on top, trading the length of a representation against how much it holds.