Learning Multimodal Embeddings with Evidence-Aligned Readout

Learning Multimodal Embeddings with Evidence-Aligned Readout. By Zirong Chen and ten others, from the Hong Kong University of Science and Technology, Guangzhou, Tencent Yuanbao, and four other institutions. Published on the twenty-seventh of September. No code repository is linked to it.

Multimodal large language models can expose task-relevant evidence by generating it. For retrieval, though, producing useful evidence is only half the question. It does not by itself determine how that evidence enters a retrieval embedding, the single vector that is indexed and scored. The authors ask whether the semantic organization of the evidence can also specify where representations are read from. Generation can make the right content appear. Whether that content shapes the vector, and from which positions it is read, is a separate decision.

EviAlign couples two pieces inside one shared multimodal large language model: Semantic Evidence Generation, and Boundary Readout. The evidence is organized into five semantic units. The model reads the contextualized state at each unit boundary, and aggregates those states into a single normalized embedding. The structure of the evidence doubles as the map of where to read.

Generation objectives and contrastive retrieval objectives train this shared structure jointly. The analysis then pulls organization and readout apart. With the same trailing readout, semantic evidence and free-form chain of thought give nearly identical retrieval performance, which suggests that evidence organization alone does not explain the full gain. Put another way, structured evidence read from the end of the sequence does no better than unstructured reasoning read the same way. A controlled two by three study follows. It compares consistent against permuted evidence organization across three readout strategies, using training targets with matched evidence spans.

With five readout states and the same mean pooling, the advantage of consistent semantic organization is zero point six five points when states are read at length-based training positions. Read at the evidence boundaries instead, the advantage grows to two point three nine. The difference between those is a one point seven four point co-design interaction. That interaction is the number to hold on to: it says organization and readout position help more together than either does apart. Across twelve MMEB retrieval tasks, EviAlign reaches seventy-six point nine average Recall at one, trained on five hundred thousand pairs, while retaining single-vector indexing and scoring.

The result is about pairing. Organizing the evidence well does little on its own, and boundaries only exist because the organization defines them. The gain appears when the two are designed together, which is exactly what the interaction term measures, and it arrives without giving up a single vector per item.