SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions

SnapBench. Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions. By Zirong Chen and nine others, from the Hong Kong University of Science and Technology, Guangzhou, Tencent Yuanbao, and three other institutions. Published on the thirtieth of August. Its code repository is public and real: eleven source files.

Point a phone at something, take a picture, and ask what it is. That snap-and-ask pattern has become one of the most common ways people enter mobile AI, which here acts as a kind of visual oracle. Real inputs are messy, though. Photos are often blurry, and the typed questions are short or mistyped. Existing benchmarks either test only on clean inputs, or do not isolate paired robustness, meaning how the image and the question hold up together under damage.

SnapBench measures exactly that. It is described as the first paired benchmark for robust snap-and-ask multimodal retrieval. The question it answers is how much retrieval degrades when the picture, the text, or both are corrupted, and whether combining the two modalities helps or hurts under noise.

It spans one thousand one hundred and forty-five queries against a gallery of nine thousand and eighty-five items. Inputs are put through fifty-three controlled corruption conditions, and the benchmark carries human annotations. Sixteen multimodal retrievers are evaluated, covering dual-tower encoders and vision language models used to produce embeddings.

Image corruptions substantially degrade retrieval. Text corruptions mostly hurt text-only retrieval and have limited impact when image and text are used jointly. And clean image-only retrieval often beats joint retrieval. The authors read that as two problems at once: a coarse text query drags the combined result down, and current systems lack a cross-modal fallback when one input is noisy.

Alongside the benchmark they propose MOOR, short for Modality-anchored, Outlier-aware, Optimal Reweighting. It is a simple adaptive fusion approach, offered to highlight the need for calibrating each modality by how reliable it is.

Adding a second modality is usually pitched as pure gain. These results complicate that. When one input is unreliable, fusing it in can cost more than leaving it out, so a retriever has to know which input to trust before it combines them. SnapBench gives that question a controlled place to be tested.