EmbeddingGemma 2: Unified and Efficient Multimodal Representations

EmbeddingGemma 2. Unified and Efficient Multimodal Representations. By Henrique Schechter Vera and seventy-two others, from Google. Published on the twenty-seventh of October.

An embedding model turns inputs into points in a space where similar things land close together. Covering many kinds of input at once is hard: quality has to hold up within a single modality, across modalities, and on inputs that mix them.

EmbeddingGemma 2 is an open multimodal embedding model that natively represents text, images, audio, video, and combinations of these in one unified space. The word natively matters. Every one of those inputs, alone or mixed, is placed in the same space by the same model, so a piece of text, a clip of audio and a video can be compared with each other directly.

Two improvements are named. One is a simplified form of per-layer embeddings. The other is a multi-stage training curriculum that progresses through modalities and through objectives. Together, the authors say, these keep quality strong on unimodal, cross-modal and mixed-modal evaluations. The design is also meant to bend to tight resources. Encoders for individual modalities can be loaded selectively, the budget of vision tokens is adjustable, embedding dimensions can be truncated, and quantized weights are offered.

On comprehensive embedding benchmarks spanning all the modalities, EmbeddingGemma 2 sets a new Pareto frontier, meaning no model in the comparison offers better quality at its size. It outperforms specialist models built for particular modalities, and it outperforms models two to twelve times its size. Ablations analyse the key design decisions.

The model is released to support further research. Its pitch rests on two things holding at once: one space for every modality, and a footprint that can shrink to fit the hardware at hand.

EmbeddingGemma 2. Unified and Efficient Multimodal Representations. By Henrique Schechter Vera and seventy-two others, from Google. Published on the twenty-seventh of October. This is the long version: the whole paper rather than the abstract.

Text embedding models are the workhorse of search and retrieval, but they only read text. To search audio or images with one, you first run speech recognition or an image captioner, which adds cost and throws away fine visual detail and non-verbal sound. Models in the CLIP family add a vision tower, but cannot take a mixed input, an image and a sentence together, as one thing. Adapting a multimodal language model into an embedder works, but those models usually carry several billion parameters, which rules out latency-sensitive or on-device use. EmbeddingGemma 2 is the attempt to get all of it into seven hundred and forty-four million parameters: text, images, audio, video, and any interleaving of them, mapped into one space.

Three things need to be in place first. The first is what an embedding is here. Inputs pass through a bidirectional transformer, the output tokens are averaged into one vector, projected to seven hundred and sixty-eight dimensions, and normalized to unit length. Similar things land close together. The second is that the model is modular. A backbone of two hundred and seventy-one million parameters is shared; a vision encoder of a hundred and sixty-seven million and an audio encoder of three hundred and five million convert media into soft tokens that enter the backbone alongside ordinary text tokens. Load only the encoders you need. The third is the teacher. The student learns from Gemini Embedding 2, a much larger model, and the backbone starts from a pretrained Gemma 4 decoder with its causal mask simply switched to bidirectional, with no separate adaptation stage.

Training runs as a curriculum in three stages, about two point nine trillion tokens in all. First, text pre-finetuning on roughly three hundred billion tokens of unsupervised query and target pairs, almost entirely distillation loss. The authors found a text-only stage important for final text quality, and frame it as an implicit warm-up for bidirectional attention. Second, multimodal pre-finetuning on about two trillion tokens of images, audio, video and documents, with distillation alone, to inherit the teacher's shared semantic space. Third, finetuning on about six hundred billion tokens, where distillation is replaced completely by a contrastive loss. That loss pulls each query toward its positive and pushes it away from hard negatives, other examples in the batch, and also same-tower negatives: other queries and other targets, which stops queries and targets from settling into disjoint regions. A spread-out regularizer pushes random pairs toward orthogonality so the embeddings fill the sphere. Matryoshka losses train shorter prefixes of the vector to work on their own, down to one hundred and twenty-eight dimensions.

Two smaller choices matter. Gemma models use per-layer embeddings, a side input to every layer drawn partly from a dedicated lookup table. That table has nothing to say about an image or a sound, so EmbeddingGemma 2 drops it and derives each layer's input by projecting the shared input embeddings, gated by the layer's own output. And the vision encoder is trained through both multimodal stages, while the audio encoder is taken from Gemma 4 unchanged and kept frozen. At the end, the final checkpoint is averaged with a much earlier one from the same run, which the authors found reduces degradation on tasks that overfit during finetuning.

The ablations are where the design is actually tested, all on MMEB-v2. Continuing distillation for a compute-equivalent stretch moves the overall score from fifty-two point two seven to fifty-two point two eight: a plateau. Switching to the contrastive finetuning recipe instead reaches fifty-five point four six, with the gains coming in images and video while visual documents stay roughly flat. Freezing the vision encoder drops the score from fifty-two point two seven to forty-eight point one eight. Dropping the per-layer lookup table helps rather than hurts, and the authors report similar gains even on text, where the table might be expected to matter most. Vision token budgets behave differently by modality: documents keep improving far beyond the training budget, while video gets worse when frames go past two hundred and eighty tokens, likely because very long sequences dilute the mean-pooled vector.

On the full benchmarks, code is the standout: seventy-eight point six eight on MTEB Code, against sixty-eight point seven six for the first EmbeddingGemma, beating every model under four billion parameters. Audio gives the clearest test of the native-modality thesis. On spoken-query retrieval, transcribing first and embedding text scores fifty-seven point five nine; feeding audio directly scores sixty-nine point five four, and even the two hundred and fifty-six dimension native embedding beats full-size transcription.

The paper has no limitations section, only future work, but limits are visible in what it reports. Every ablation uses an intermediate recipe, which the authors say is not representative of the final model, and the ablations evaluate on MMEB-v2 only, which excludes audio entirely. The claim that training the audio encoder would bring similar gains is a conjecture, not a result. English text regresses slightly against the predecessor. On MAEB the mean task score sits just below Jina's omni nano model. No error bars or repeated runs are reported, and most baseline scores are taken from public leaderboards. Finally, the quantized vision and speech numbers were measured with a different evaluation harness from the float ones, so those gaps are not a clean comparison.

The idea to carry away is objective progression. Distillation is the fastest way to inherit a large teacher's map of meaning across modalities, but it plateaus, and the student only becomes better than a copy when it stops imitating and is trained on its own contrastive targets. Pair that with modular encoders, truncatable vectors, and int4 weights that bring the full multimodal model to five hundred and sixty-seven megabytes of peak memory on a Pixel phone, and a single embedding space for text, sight and sound becomes something a device can carry.