Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

Chaos in the Text. Revealing the Modality Preference in Mixed-Modality Retrievers. By Yubo Sun and eight others, from the University of the Chinese Academy of Sciences, Northeastern University, and two other institutions. Published on the eighth of October.

Dense retrievers have advanced on text corpora and on image corpora. Whether that holds up on mixed corpora, which contain text documents, image documents, and fused text-image documents together, has been unclear. The authors' claim is that it does not hold reliably. Retriever performance is highly sensitive to the modality composition of the corpus. And the sensitivity is lopsided: irrelevant text causes more severe degradation than the same number of irrelevant images. They call this Chaos in the Text.

The evidence comes from a systematic study of retrievers across architectures. Image documents were progressively replaced with text representations that correspond to them semantically. Retrieval performance traces a pronounced V-shaped curve. It stays strong when the corpus is all one modality, and degrades substantially when modalities coexist.

Further analysis points to a cause, which the authors name modality preference. Text representations receive systematically higher similarity scores. That lets irrelevant text outrank relevant images, which explains why text distractors are so damaging.

To mitigate the bias they introduce Trident. It constructs a text view, an image view and a fused text-image view of each document, and treats all three as co-equal positives. Training jointly optimizes relevance discrimination and balance across the positive views, through a Multi-Positive View InfoNCE objective. On visual document benchmarks and natural image benchmarks, Trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures. It reduces sensitivity to modality composition and to text distractors, and it also raises average single-modality retrieval performance.

The finding concerns dense retrievers searching corpora where modalities are mixed. On single-modality corpora the same retrievers do well, so the problem appears specifically when text and images compete in one index. The fix is reported for CLIP-based and VLM-based retrievers, on visual document and natural image benchmarks.

A retriever that looks fine on a clean benchmark can quietly favor one kind of document once the corpus is mixed. Here the favored kind is text, and the consequence is that an irrelevant paragraph can beat the right picture. Trident's remedy is to train every document as three equal views, so no single modality gets an edge by default.