Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Ovis-Embedding. Pushing the Frontiers of Universal Omni-Modal Embeddings. By Embedding Team, from Alibaba Group. Published on the twenty-first of September. Its code repository is a placeholder: a readme, and nothing to run yet.

An embedding model that serves text, images, video and audio is often assembled from separate towers, one per modality. That arrangement fragments the space the embeddings live in. Each tower learns its own representation, and matching across them has to be arranged after the fact. The report calls this modality fragmentation, and sets out to avoid it from the start.

Ovis-Embedding is an omni-modal embedding family built on native integration of all four. Instead of separate towers, one shared multimodal backbone encodes every modality into a common representation space. The report describes three advances that make this work: how the model is initialized, what data it is trained on, and how training and inference are tuned for the specific job of producing embeddings.

The first is native omni-modal initialization. A pretrained Qwen-omni model becomes the embedding backbone, adapted through contrastive training with low-rank initialization. The second is data-centric training. The authors build a broad, high-quality corpus spanning text, images, video, audio, and interleaved multimodal data. To use it efficiently, they introduce homogeneous-source sampling, which forms batches that are consistent in task, so the negatives within each batch are informative. The aim is that the other examples in a batch are ones the model can actually learn from when it tells them apart.

The third advance is optimization specific to embeddings, at training and at inference. Focal loss puts emphasis on hard examples. A similarity-based Embedding Distillation transfers fine-grained similarity structure from complementary experts. At inference, low-rank feature decomposition yields compact embeddings with flexible dimensionality, at minimal cost in performance.

In empirical evaluations, the Ovis-Embedding family reaches state-of-the-art performance on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB. Together those cover text, image, video and audio. The spread is the point. A single family performing at that level across all four modalities is the evidence that one shared backbone can serve them all.

The report's argument is that fragmentation across modalities can be overcome by training one model on all of them from the start. The authors see unified omni-modal training as a way toward universal embedding models for any-to-any retrieval, where a query in one modality can find matches in any other.