Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models. By Siting Li and four others, from the University of Washington and Amazon FAR. Published on the eighth of September. Its code repository is worked out: about six hundred source files, with a readme and a licence.

An image tokenizer defines the visual language of a unified multimodal model: the vocabulary in which images are written, alongside text. Tokenizers are usually studied through isolated metrics, or through evaluations of generation alone or understanding alone. The authors argue those evaluations miss how visual tokens behave when they are modelled jointly with text, and they propose to study tokenizers through the training losses of that joint modelling. Their findings come in four parts.

First, losses should be analysed task by task, because different tasks show distinct scaling behaviour and rank tokenizers differently. Second, the relationship between loss and performance depends on the token space being predicted. Third, better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance. Fourth, the choice of image tokenizer can affect how well text itself is modelled under joint optimization. Put together, the four findings say that the usual ways of scoring a tokenizer can point in a different direction from how it actually behaves once text is in the same model.

The evidence comes from a controlled, purely autoregressive testbed. During multimodal continual pretraining, the authors track validation losses for four prediction tasks: text, image, text-to-image, and image-to-text. They examine how those losses scale and how they relate to downstream performance, then use them to study multimodal learnability, meaning how well image and text tokens are jointly modelled, and to study tokenizer design.

The second finding carries the most detail. With a fixed tokenizer, both the text-to-image loss and the image-to-text loss correlate with generation quality. Across different tokenizers, though, the text-to-image relationship shifts with the image-token space. The image-to-text loss is computed over a shared text vocabulary, so it gives a more consistent signal. It also correlates with both generation and visual understanding performance after supervised fine-tuning.

As case studies, the authors revisit three axes of tokenizer design: the discriminator, semantic supervision, and vocabulary size, examining how each affects joint modelling and downstream performance. In each case the question is the same one the testbed was built to ask: not how the choice changes the tokenizer in isolation, but how it changes the model that has to learn images and text together.

The claims are made within one controlled, pure-autoregressive testbed, trained through multimodal continual pretraining. The authors present it as a complementary perspective on tokenizers, alongside the usual metrics rather than in place of them. The design axes examined are the three named, not every choice a tokenizer involves. The findings about loss are also findings about correlation: losses are offered as a lens on downstream performance, tracking it under the conditions described, rather than as a replacement for measuring it.

The picture is of a tokenizer as a language that text has to share a model with. Judged alone, by how well it reconstructs an image, a tokenizer can look better than it is. Judged by how it trains beside text, and especially by the image-to-text loss on a shared vocabulary, it can be compared across designs on common ground.