What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

What Generation Knows. Unlocking Fine-Grained Understanding in Unified Multimodal Models. By Tengju Ru and eight others, from Zhejiang University, the University of Hong Kong, and two other institutions. Published on the fourteenth of October. Its code repository is public and real: about thirty-five source files.

Fine-grained visual understanding means telling apart subtle differences in local attributes and in spatial relations, and existing vision-language models still find it hard. Unified multimodal models offer a new route, because they contain a generation pathway that preserves rich visual information, complementary to what the understanding side holds. But methods that use generation to help understanding have mostly relied on generative supervision or on explicitly generating images. Using the native representations on the generation side directly has not been systematically explored. The information is already inside the model, sitting on the generation side, and nobody has looked carefully at reading it from there.

The authors begin with an analysis rather than a design, asking what those generation-side representations actually offer for fine-grained understanding. Three findings come out of it. First, generation-side representations are more sensitive to fine-grained semantic changes. Second, they capture task-relevant changes that the understanding-side representations miss altogether. Third, they depend on correct spatial organisation for that advantage: the benefit is tied to how the information is laid out, not only to what it contains. Each finding shapes the framework that follows.

Guided by those findings, they design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. GAFU compresses the generation-side visual representations into a small set of compact tokens, which the authors call G tokens, designed to keep fine-grained visual detail. The understanding pathway can then retrieve from those tokens, conditioned on the task in front of it. Instead of generating an image and looking at it, or training with a generative objective, the understanding side reads directly from what the generation side already holds, and takes only what the task calls for.

The comparison is kept fair on data. GAFU consistently outperforms supervised fine-tuning baselines trained on exactly the same data, so the gains come from how the model uses its own generation side rather than from extra training material. This holds across diverse visual understanding tasks and across different architectures of unified model. The maximum relative improvements are twenty percent on VStar-Bench, seventeen point nine four percent on OmniSpatial, and eight point six two percent on VSR-Bench.

The argument runs from a measurement to a design. First establish that the generation side notices things the understanding side does not, then build a narrow channel that lets understanding ask for them. The authors frame it as a path toward real synergy between generation and understanding inside one model, rather than two abilities that merely share a network.