Factorized MLLM Embeddings for Composed Image Retrieval
WonJun Moon, Seongchan Kim, Dahun Kim, Seungryong Kim
KAIST AI · Google DeepMind
Published November 3, 2026 · on the feed November 4, 2026 · ▲ 48 · arXiv 2611.05193
The abstract
Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized. This stems from directly mapping composed queries into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. In this work, we introduce Factorized MLLM Embeddings (FactEmbed), which decouples compositional query comprehension from building discriminative embeddings. Rather than directly projecting the composed query into a single embedding without internal reasoning, FactEmbed employs a set of factorized latents that simulate the intended visual modifications via an intermediate target imagination objective. This process encourages the latents to ground into target semantics before forming discriminative representations for retrieval. Subsequently, to propagate these factorized representations throughout the model for retrieval, we optimize a factor-level contrastive loss while distilling the decomposed representations into a deeper attention layer of the MLLM. Finally, for efficient inference, we retain the [EOS] token to establish a global-level embedding on top of these factorized latents, while we exploit the mean-pooled factorized latents to rerank a short candidate list at negligible cost, as both are produced by the same forward pass. Extensive evaluations demonstrate that FactEmbed outperforms prior works across CIR benchmarks.
The Brief
2:28A short podcast summary of the paper. Written and voiced by AI.
Transcript
Factorized MLLM Embeddings for Composed Image Retrieval. By WonJun Moon and three others, from KAIST AI and Google DeepMind. Published on the third of November.
Composed Image Retrieval, or CIR, asks a system to find a target image given two things: a reference image, and a text describing a modification to it. The answer is neither image the user supplied. It is some third image that looks like the reference once the requested change has been applied, which means the system has to work out that change before it can search for it. Recent approaches built on Multimodal Large Language Models have made significant progress. The authors argue, though, that the compositional understanding of the MLLM remains underused. The cause they point to is the training path. Composed queries are mapped directly into a retrieval embedding using contrastive supervision alone, and nothing explicitly resolves which visual modification the query actually intends.
Factorized MLLM Embeddings, called FactEmbed, decouple two jobs that this setup merges: comprehending the composed query, and building a discriminative embedding. Instead of projecting the query straight into a single embedding with no internal reasoning, FactEmbed uses a set of factorized latents. Those latents simulate the intended visual modifications through an intermediate objective of imagining the target. The aim is for the latents to ground into target semantics first, and only then form the representations used for retrieval.
Two training signals carry these factorized representations through the model. A contrastive loss is applied at the level of individual factors, and the decomposed representations are distilled into a deeper attention layer of the MLLM. Inference is built for efficiency. The EOS token is kept to provide a global-level embedding on top of the factorized latents. The factorized latents themselves, mean-pooled, are used to rerank a short candidate list. Because both come out of the same forward pass, that reranking costs a negligible amount.
The evidence in the abstract is stated broadly. Extensive evaluations show FactEmbed outperforming prior works across CIR benchmarks. The specific benchmarks and margins are not given there, so they are not given here either. What the claim does cover is the whole design at once: the factorized latents, the imagination objective, the distillation, and the two-stage retrieval with reranking, measured together against earlier approaches.
The bet is on order of operations. Resolve the modification first, as an explicit intermediate, and only then compress it for retrieval. And the same latents that do the resolving end up doing the reranking as well, which is why the second stage arrives almost for free.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a search that never stops to picture what you asked for.
You are building a search that takes a photo plus a text like 'same dress, but cream with animal print' and must find a third image: the photo with that change applied.
The road ahead
- 1 · The problem ~7 min Searching with a photo plus an edit request means finding a third image, and current systems never work out the edit before they search.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Add learnable tokens that first imagine the edited target, make them useful for search, then use them both to build and to rerank results.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Insert tokens, train them to imagine the target, align them for search, distill deeper, then retrieve with EOS and rerank with their mean.
- 6 · The experiment ~6 min You’ll guess: Reranking with the factor tokens comes on top of the usual single-vector search. How much does that reranking cost at inference?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.