Can Visual Generation Expand What Unified Multimodal Models Know?

Can Visual Generation Expand What Unified Multimodal Models Know?. By Wenkai Li and three others, from Tsinghua University and Z dot AI. Published on the fifteenth of October. Its code repository is public but slight: three source files.

Unified multimodal models, or UMMs, are trained to generate images, and that supervision exposes them to visual information that text does not fully document. The question here is whether learning to generate images can expand what a pretrained model knows, in a form it can then use when answering in words. The authors' answer is yes, directly, but unevenly.

To test it they introduce Knowledge Transfer Probing, or KTP. It is a controlled protocol: new knowledge is supplied only through image-generation training, and whether that knowledge is accessible is then tested through text-only question answering. Across seven knowledge categories and three UMMs, knowledge does transfer from visual supervision to language, though not equally everywhere.

Building on that, the authors add two alignment steps in turn. Textual Alignment connects generation and answering through shared knowledge. Interleaved Image Text Alignment connects them through interleaved responses. Guidance on a selected set of entities improves knowledge access on other entities, without any QA supervision, and interleaved alignment adds further gains. A further analysis finds that knowledge a model reproduces correctly in its images is often more accessible through text as well.

The transfer is partial. It is uneven across categories, and even where an image shows the knowledge correctly, substantial gaps remain between getting at it through generation and getting at it through text. The study covers three models and seven categories. The authors do not present the problem as solved: they argue that making this knowledge broadly accessible will take joint work on training data design and model architecture.

Image generation, on this evidence, is a channel for learning and not only a skill to be learned. What a model picks up while drawing can surface when it is asked a question in text, at least part of the time, and that makes visual generation a source of knowledge beyond what textual supervision supplies.

Can Visual Generation Expand What Unified Multimodal Models Know?. By Wenkai Li and three others, from Tsinghua University and Z dot AI. Published on the fifteenth of October. This is the long version: the whole paper rather than the abstract.

A caption reads: a map of Westeros. The image it describes shows where the coastlines run and which region lies southeast of which. None of that is in the caption. A unified multimodal model, one network that both understands and generates images, is trained to draw that map from that caption, so it has been supervised on facts no text ever stated. The question here is whether those facts become knowledge the model can use in words. Ask it, with no image in front of it, where the Stormlands sit relative to the Westerlands. Being able to draw the answer does not guarantee being able to say it. Earlier work named that failure Modal Aphasia: models that reproduce a learned visual concept but struggle to describe it.

To test that cleanly, the authors build a protocol called Knowledge Transfer Probing. The unit of knowledge is a triplet: an entity, an attribute, and a value. Arvenbrock's jersey, its number, ten. Every entity is invented and lives in a fictional setting called MagicWorld, so pretraining cannot already supply the answer. Training prompts name the entity and describe a scene but never state the value; the value appears only in the target image. The test is a plain text question with no image: what number is on Arvenbrock's jersey? The share answered correctly is called textual accessibility.

There are seven categories of fact: color, count and shape; spatial arrangement and relative size; and two kinds of written content, a jersey number and a book title. Each category has fifty triplets, each rendered in about twenty-five varied scenes. Three models with different designs take part. BAGEL has separate understanding and generation branches. SenseNova-U1 models pixels without a separate pretrained visual encoder. GLM-Omni, an in-house model, turns images into discrete tokens.

The study runs three conditions. Direct transfer is image training alone. The other two split each category's entities in half, into an alignment set and a test set. Every entity still gets image training, but alignment entities also get supervision linking their facts to answers. Test entities never receive question and answer supervision, so any gain on them means the model has learned to reach visually supplied facts in general, not that it memorised answers.

Textual alignment gives each alignment entity ten text-only examples: a question asking for the attribute, brief reasoning, then an answer. Interleaved image and text alignment changes how that example is organised. The question ends with a request to think with images, and the response itself contains a generated image. The model recalls the entity by drawing it, observes what it drew, and answers. Generation, observation and answering sit inside one training sequence instead of separate examples. At test time the question is still text only, with no image generated along the way.

A second measure sits beside the first. Visual accessibility asks the model to generate the entity from one of its training prompts, and a vision-language model judges whether the picture shows the right value. The difference between the two is the accessibility gap: how much the model can draw but cannot say.

Direct transfer works, but barely. Averaged over the seven categories, BAGEL answers eight point eight six percent correctly and GLM-Omni fourteen point five seven, both below the random-guessing average of fifteen point nine six. SenseNova-U1 reaches twenty percent overall and sixty-eight on relative size. Visual accessibility is far higher for every model: SenseNova-U1 draws the right jersey number eighty-eight percent of the time and states it zero percent of the time. Alignment then helps on entities it never touched. Textual alignment lifts BAGEL and SenseNova-U1 but lowers GLM-Omni. Interleaved alignment beats textual alignment on average for all three, and GLM-Omni gains most.

The comparison that most directly tests the thesis is an ablation on GLM-Omni. Keep the interleaved sequences and their text supervision, but mask the loss on the image portion. Average accuracy falls from twenty-six point eight six percent to thirteen point seven one. The benefit is not the format of thinking with images. It is being trained to actually generate the image inside the reasoning.

The written categories resist almost everything. On test entities, all three models score zero on jersey numbers and book titles after either kind of alignment, while the alignment entities themselves reach seventy-two percent on numbers. Letting the model draw before answering at test time, which the authors call visual chain-of-thought, finally moves numbers off zero, as high as thirty-six percent for GLM-Omni.

There is no limitations section. The conclusion lists future work instead: natural visual data, isolating architectural effects, and better alignment data and inference strategies. Every fact is synthetic and invented, so whether real photographs teach this way is open. The three models differ in architecture but also in training steps and data mixing ratios, so cross-model differences cannot be pinned on architecture alone. The image-loss ablation was run on GLM-Omni only.

Each category's test set is twenty-five entities, scored once with no error bars, so a single entity moves a category score by four points, and GLM-Omni's jump on spatial arrangement from zero to forty percent is ten entities.

The idea to keep is that a unified model can learn facts from images that no text ever told it, and some of them do reach language, but the route is not automatic. What carried more of them across was a training example in which the model draws, looks, and then answers, with the drawing itself supervised. Generation is a source of knowledge. Whether it becomes something the model can say depends on how the training data ties the two together.