HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents
Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri, Seung Hun Eddie Han, Ryandito Diandaru, Qinrong Cui, Jan Christian Blaise Cruz, Badrinath Chandana and 9 others
Mohamed bin Zayed University of Artificial Intelligence · Mila – Quebec Artificial Intelligence Institute · Inception AI · Alibaba Group · AI Singapore
Published October 2, 2026 · on the feed October 5, 2026 · ▲ 60 · arXiv 2610.03574
The abstract
We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native or highly proficient speakers. Questions are designed to be extremely challenging. Each question targets a concise, publicly verifiable answer whose discovery requires locating obscure evidence, following multi-step clue chains, or inspecting heterogeneous sources such as videos, scanned documents, images, or maps. Easier questions are filtered out by evaluating them with models without internet access to reduce the likelihood that they can be answered with parametric knowledge alone. We evaluate several models using provider-native search and a shared external retrieval harness under a common agent protocol. To contextualize model performance and effort, we also conduct a human evaluation on a sample of the questions. HyperBrowseComp provides a challenging testbed for persistent information seeking across languages and evidence modalities, with difficulty arising from discovering and connecting evidence on the open web.
The Brief
1:57A short podcast summary of the paper. Written and voiced by AI.
Transcript
HyperBrowseComp. A Multilingual and Multimodal Stress Test for Web-Browsing Agents. By Alham Fikri Aji and sixteen others, from the Mohamed bin Zayed University of Artificial Intelligence, Mila, the Quebec Artificial Intelligence Institute, and three other institutions. Published on the second of October. No code repository is linked to it.
A web-browsing agent is supposed to find things out, persistently, across whatever the web holds. The abstract frames the test it needs as one where the difficulty comes from discovering evidence and connecting it, across languages and across kinds of evidence, rather than from recalling what a model already knows.
HyperBrowseComp is a browsing benchmark of four hundred and twenty-three questions in thirteen languages. Every question was written by hand by a native or highly proficient speaker, and validated by humans. Each one targets a concise answer that can be publicly verified, but getting to it means locating obscure evidence, following a chain of clues over several steps, or inspecting heterogeneous sources: videos, scanned documents, images, or maps.
The questions are meant to be extremely challenging, and there is a filter to keep them that way. Each was tried on models with no internet access, and the ones those models could answer were removed, which reduces the chance that a question can be solved from parametric knowledge alone. Models are then evaluated in two ways under one common agent protocol: with the search their provider offers natively, and with a shared external retrieval harness. A human evaluation on a sample of the questions sits alongside, to put model performance and effort in context.
The abstract does not report scores. What it states is the result of the construction: a testbed in which the difficulty arises from finding and linking evidence on the open web, across languages and evidence modalities.
The filter is the design choice carrying the most weight. By discarding anything a model could answer offline, HyperBrowseComp tries to ensure that a correct answer is evidence of browsing, and the human sample gives a sense of how hard that browsing is for people too.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free