VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents

VisSearch-Bench. Evaluating Long Horizon Visual Search in Multimodal Agents. By ZhengXian Wu and twelve others, from Tsinghua University, Ant Group, and one other institution. Published on the third of November.

Multimodal agents call web and image tools to answer questions that lie beyond their internal knowledge. Many existing search tasks, though, use the image mainly to identify a starting entity, and from there the work is text retrieval. Tasks like that reveal little about how an agent connects information across images, across objects, and across search steps. Once the image has pointed to a name, the rest of the task could almost be done without it, so success says more about text search than about visual reasoning. A benchmark that wants to probe the visual side has to keep the images relevant for the whole search.

The target here is visual context management: organising visual information, using it, and linking it to other findings as a search unfolds. Visually intensive tasks demand exactly this. An agent has to combine an object's appearance, its local details, or spatial relations with information it finds externally.

VisSearch-Bench contains two hundred and four human-written tasks, spread over seven main domains and twenty-five subdomains. They fall into five kinds of search: Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Every task comes with a reference answer, traceable evidence, and scoring rules. Two metrics read the results. Accuracy counts complete task success. Process Score measures how correct and complete the individual results and subquestion answers are, so partial progress is still visible.

Sixteen multimodal models were evaluated. The best system fully solves only twenty-six point four seven percent of the tasks, and that is with search tools available. Roughly three tasks in four remain unsolved by the strongest model tested. The authors read this as a sign of how hard it is to link visual evidence with search results and produce a complete answer.

A low ceiling like this one locates where the difficulty sits. Finding pages is not the obstacle the benchmark exposes. Holding on to what was seen in an image, and tying it to what search turns up, is. Pairing all-or-nothing Accuracy with a Process Score means future systems can be credited for getting part of that chain right, while the headline number keeps the standard at a complete answer.