Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

Multimodal Flow. Unified Flow Modeling of Language and Vision in Embedding Spaces. By Hongyuan Tao and nine others, from the Huazhong University of Science and Technology, Beijing Jiaotong University, and one other institution. Published on the thirtieth of September. Its code repository is worked out: about one hundred and ten source files, with a readme and a licence.

Unified models of language and vision have mostly taken one of two routes. One treats both language and quantized images as discrete tokens, and that introduces a visual quantization bottleneck. The other combines discrete prediction for language with continuous generation for images, and that needs different objectives and sampling procedures for each modality. Modelling everything continuously avoids both trade-offs and allows one shared generative process. For multimodal pretraining, though, that path remains underexplored. Each of the two familiar routes, in other words, pays for unification with something. The first pays with fidelity on the image side, squeezed through a quantizer. The second pays with complexity, carrying two kinds of training and two kinds of generation inside one model.

Multimodal Flow is a fully continuous generative model of language and vision. Its architecture integrates continuous representations of both modalities with a single backbone, a shared chunk-causal flow model. Text and images are not handled by separate machinery. They are organized into one ordered sequence of continuous units the authors call hyperchunks.

Text blocks and images become ordered continuous hyperchunks, in a way that preserves the order of textual tokens and the spatial structure of images. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint attention lets the modalities interact, while modality-specific feed-forward networks process each one. So the modalities share attention, and with it the ability to inform each other, while each keeps its own feed-forward processing. Training and generation also differ in rhythm. During training the model predicts several target chunks in parallel. At inference it generates hyperchunks one after another, the chunk-causal order the backbone is built around.

The authors instantiate MF-1 and pretrain it on multimodal data, at sizes of zero point six billion, one point two billion, and one point six billion parameters. At each scale, continued pretraining consistently improves multimodal modelling.

With only one hundred and fifty billion pretraining tokens, MF-1 averages eighty-two point eight across GenEval and DPG-Bench for generation, and seventy-five point three across VQAv2, MMBench and POPE for understanding. That stays competitive with unified models trained on substantially more data. Under matched data, optimization and parameter budgets, Multimodal Flow also outperforms representative hybrid and discrete models. That matched comparison sets the continuous approach against the two routes it set out to replace, hybrid and discrete, on equal terms.

Strip the design down and one choice remains: nothing is discrete. Images are not quantized, and language is not predicted token by token through a separate objective. Both live as continuous embeddings, ordered into hyperchunks, and one vector field governs words and pixels alike. The authors propose this continuous, chunk-based embedding flow modelling as a new, fully continuous paradigm for unified multimodal modelling. Code and model are publicly released.