Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation

Omni-Embed-Mini. Binding Modalities Without Forgetting via Dense Distillation. By Mohammed Irfan Kurpath and four others, from the Mohamed bin Zayed University of Artificial Intelligence. Published on the first of October. Its code repository is worked out: about sixty-five source files, with a readme.

Teach a text embedding model to handle new modalities and its text retrieval usually gets worse. The omni-modal embedders that exist today make up for that loss with size, running to multiple billions of parameters. Covering more kinds of input has meant either giving up text quality or paying for it in scale. The question behind this work is whether a small model can take on speech, sound, pictures and documents while the part that handles text is left exactly as it was.

Omni-Embed-Mini is a model of zero point nine billion parameters that maps text, speech, audio, images, video and visually-rich documents into one shared cosine space, and it does so without updating a single text-side parameter. The key observation concerns where the teacher signal comes from. No separate embedding model is needed. Each media sample is paired with a dense cascaded caption, and the training target is simply the frozen backbone's own embedding of that caption. Teacher and student share the same backbone weights, so they live in byte-identical geometry. A clip of audio or a page of a document is, in effect, asked to land where a careful description of it would already land.

Because the geometry is already shared, alignment needs only light machinery: small projectors, plus LoRA adapters on the modality encoders, introduced in phases. Training combines a Matryoshka SigLIP contrastive loss with an online hybrid miner of hard negatives, whose negatives sharpen as the encoder improves. The same recipe carries over to a larger variant of two point three billion parameters, where a native vision-language backbone is swapped in. Nothing else about the approach has to change for that move, which is what makes it a recipe rather than a single model.

Since the smaller model keeps its text weights bit-identical to the backbone, training cannot regress text retrieval at all. It holds forty-nine point five seven nDCG at ten on the BEIR-8 portion of MTEB-v2, while extending to five additional modalities. It is also roughly two point seven to nine point five times smaller than every open omni embedder the authors compare it with. The larger variant is competitive with the closed gemini-embedding-2, and edges ahead of it on the average across all modalities.

Forgetting, the problem in the title, is ruled out by construction rather than treated as a risk to be kept small during training. Freeze the text side, let the backbone grade its own captions, and the new modalities have to come to the text space instead of the other way round. Models, code, data and the evaluation harness are released on the project page.