Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
Hangrui Xu, Zhengxian Wu, Yunyao Yu, Zhuohong Chen, Rui Cong, Xiangwen Deng, Zhifang Liu, Peng Jiao and 1 others
Tsinghua University · University of Arizona
Published August 19, 2026 · on the feed September 3, 2026 · ▲ 26 · arXiv 2608.21450
The abstract
Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
The Brief
2:18A short podcast summary of the paper. Written and voiced by AI.
Transcript
Beyond Visual Similarity. Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering. By Hangrui Xu and eight others, from Tsinghua University and the University of Arizona. Published on the nineteenth of August. Its code repository is public and real: nineteen source files.
Knowledge-based visual question answering asks about things in an image that a model cannot answer from the picture alone, often long-tail entities, so it has to retrieve outside information first. The retrieval step is where the trouble sits. Existing pipelines mostly use CLIP-style dual encoders, and those reward surface-level visual similarity rather than agreement about which entity is actually shown. Two failures follow. The same concept can look very different from one photo to the next, and be missed. Distinct entities can look alike, and be confused.
KBMR replaces that retriever with one built on a multimodal large language model. The authors describe it as the first embedding retriever of this kind tailored to the task. It leans on the autoregressive capabilities of these models to map images into a semantic space that better preserves concept identity, so that nearness in the space tracks which thing it is, rather than what it looks like.
Training it raises a second problem. At Wikipedia scale, the supervision is noisy, and binary labels of match or no match are too rigid to describe it. KBMR adds a semantic discriminator, itself built on a multimodal language model, that produces continuous entity-consistency weights instead. Those weights drive a new objective the authors call continuous semantic distillation. Because supervision is now graded rather than yes or no, the objective supports effective hard negative sampling and soft supervision beyond rigid binary labels.
Extensive experiments compare KBMR with CLIP baselines and report that it significantly outperforms them. The gains reach up to fourteen point seven percent on retrieval Recall@1. Carried through to the final answers, the end-to-end question answering accuracy rises by up to nine point four percent. Those are best-case figures, given as upper ends of the improvement.
The argument runs in a straight line from the diagnosis. If retrieval fails because it measures resemblance, the fix is a retriever that measures identity, and a training signal that admits degrees of sameness rather than pretending every pair is a clean match or a clean miss. The code is public.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The right answer depends on finding the right page. Looks keep leading the search astray.
You are building a system that answers questions about images, like who owns the hotel in a photo. The picture alone can't answer, so you must first retrieve the right entry from a Wikipedia-scale knowledge base.
The road ahead
- 1 · The problem ~7 min To answer questions about a photo, you must first fetch the right knowledge entry, and a retriever that matches looks often fetches the wrong one.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Embed images with a multimodal language model so nearness means same entity, and train it on graded match scores from a second model acting as judge.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min A judge scores pairs, the scores become graded weights, the retriever is trained to match them, then it searches by similarity.
- 6 · The experiment ~6 min You’ll guess: At best, how much did the multimodal-language-model retriever improve retrieval Recall@1 over CLIP baselines?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.