NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
Aurélien Lac, Tony Wu
H Company
Published August 31, 2026 · on the feed September 3, 2026 · ▲ 24 · arXiv 2609.01657
The abstract
Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.
The Brief
2:51A short podcast summary of the paper. Written and voiced by AI.
Transcript
NeoMME. A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference. By Aurélien Lac and one other, from H Company. Published on the thirty-first of August. Its code repository is worked out: about eighty source files, with a readme and a licence.
Multimodal models are often built on architectures designed for generative vision-language modelling: a separately pretrained vision encoder joined to a causal language model. Visual document retrievers such as ColPali take models like that and repurpose them as encoders. Retrieval is not a generative task, yet those retrievers carry over the full parameter and compute overhead of a VLM anyway. The model being repurposed was shaped to produce text, and the retriever inherits that shape, together with its size and its cost, whether or not the retrieval task needs any of it.
NeoMME starts from the other end. It is a family of Multimodal and Multilingual bidirectional Encoders, one with two hundred and sixty million parameters and one with eight hundred million, that process multilingual text and raw image patches in a single bidirectional Transformer encoder. One tower, rather than a vision model attached to a language model.
Both models are pretrained from scratch. The text objective is masked discrete diffusion, and for multimodal examples it is conditioned on the visible image patches. Put another way, the model learns by recovering masked text, and when an image is present, what it can see of that image is part of what it recovers the text from. Both support a context of sixteen thousand three hundred and eighty-four tokens, enough to encode up to two standard four K UHD images. To show what the backbone can do downstream, the authors fine-tune it with dense and late-interaction heads trained jointly, giving NeoMME-Retriever.
On the ViDoRe v3 benchmark, the smaller NeoMME-Retriever scores zero point five two three nDCG at ten. That outperforms every evaluated model strictly below eight hundred million parameters. The larger version reaches zero point five five six. Both figures belong to the fine-tuned retrievers, not to the raw backbone.
Efficiency is measured too. At a matched image input size of two thousand and forty-eight by two thousand and forty-eight, on an NVIDIA L40S, the smaller NeoMME encodes pages with about twice the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress the late-interaction multimodal document embeddings by two hundred and fifty-five times, while preserving over ninety-five percent of baseline nDCG at ten.
The release is part of the contribution. NeoMME is contributed to Hugging Face Transformers, and the pretrained backbone and the retrieval-compatible checkpoints are released under Apache two point zero. Underneath sits a simple argument about fit: when the job is encoding rather than generating, train an encoder for it from the start, instead of inheriting the overhead of a model built to write.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Retrieval generates nothing. So why pay for a generator?
You need a retriever that finds document pages from a query. The popular ones take a generative vision-language model, a pretrained vision encoder bolted onto a causal language model, and reuse it as an encoder.
The road ahead
- 1 · The problem ~7 min Page retrievers reuse big generative vision-language models as encoders, so they pay for machinery that retrieval never uses, on every large page image.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min One small bidirectional Transformer reads text and raw image patches together, learns by recovering masked text, then serves retrieval with compact stored vectors.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Feed text and patches into one encoder, pretrain it by recovering masked text, fine-tune two retrieval heads together, then compress the stored vectors.
- 6 · The experiment ~6 min You’ll guess: Combining hierarchical token pooling with asymmetric quantization, how many times smaller did the late-interaction document embeddings become?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.