MULTI3IR: A Benchmark for Multi-perspective Multi-domain Multi-modal Information Retrieval
Seokwon Song, Sohyeon Kim, Gunhee Kim
Seoul National University
Published August 31, 2026 · on the feed September 3, 2026 · ▲ 8 · arXiv 2608.30949
The abstract
Information retrieval (IR) increasingly targets open-ended queries that admit diverse perspectives. Existing IR benchmarks, however, focus primarily on closed-ended queries, while even open-ended benchmarks largely consist of queries whose supporting documents span a single subject domain and modality. We introduce Multi^3IR, a benchmark that evaluates how well retrievers cover the multifaceted perspectives of open-ended queries across diverse domains and modalities. It comprises 104.9K Stack Exchange queries, each annotated with perspective descriptions that capture the query's implicit viewpoints. We further propose SPIN, a parameter- and label-efficient method that learns noise vectors to steer embeddings toward diverse yet meaningful semantic directions. Experiments show that existing multimodal retrievers suffer from single-perspective bias, while SPIN substantially improves perspective coverage on Multi^3IR and generalizes well to unseen open-ended IR benchmarks. The dataset and experimental code are available at https://github.com/seokwon99/Multi3IR.
The Brief
1:48A short podcast summary of the paper. Written and voiced by AI.
Transcript
MULTI3IR. A Benchmark for Multi-perspective Multi-domain Multi-modal Information Retrieval. By Seokwon Song and two others, from Seoul National University. Published on the thirty-first of August. It links to a code repository that is no longer there.
Information retrieval increasingly has to serve open-ended queries, the kind that admit diverse perspectives rather than a single right answer. The benchmarks used to measure retrievers have not kept pace. Most focus on closed-ended queries. Even the open-ended ones are mostly made of queries whose supporting documents sit within one subject domain and one modality.
Multi3IR asks a different question of a retriever: how well does it cover the many perspectives an open-ended query carries, when those perspectives are spread across diverse domains and modalities. The target is coverage of viewpoints, not just the relevance of a single hit.
The benchmark comprises one hundred and four point nine thousand queries drawn from Stack Exchange. Each query is annotated with perspective descriptions, which capture the viewpoints the query implies without stating them. Alongside the benchmark the authors propose SPIN, a method that is efficient in both parameters and labels. It learns noise vectors that steer embeddings toward semantic directions that are diverse while still meaningful. The dataset and the experimental code are publicly available.
Existing multimodal retrievers show a single-perspective bias. Given an open-ended query, they tend to settle on one viewpoint. SPIN substantially improves perspective coverage on Multi3IR, and it generalizes well to other open-ended retrieval benchmarks it had not seen.
A retriever can be relevant and still narrow. This benchmark makes that narrowness measurable, by writing down the hidden viewpoints behind each query and checking which of them the results reach. SPIN is offered as one answer, a light way of nudging embeddings apart.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Ten results, one viewpoint, and the rest of the question left unanswered.
You run search for open-ended questions, like how people perceive blue foods. A good answer draws on psychology, history and social issues, from both text and images.
The road ahead
- 1 · The problem ~7 min Open-ended questions need answers from many viewpoints, but a retriever built around one query embedding keeps returning documents that all say the same thing.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Learn a few noise vectors, add them at a middle layer of a frozen retriever, and get several query embeddings that each point at a different viewpoint.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Annotate viewpoints, learn steering vectors from them, search with each steered embedding, then merge the lists into one result set.
- 6 · The experiment ~6 min You’ll guess: On open-ended queries with several implied viewpoints, how did existing multimodal retrievers and SPIN fare?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.