Deep Search in Real-World Web Environments

Deep Search in Real-World Web Environments. By Xiaomin Yu and twenty-four others, from the Hong Kong University of Science and Technology (Guangzhou), Fudan University, and twelve other institutions. Published on the nineteenth of October. Its code repository is worked out: about forty source files, with a readme and a licence.

Deep Search agents have made substantial progress, but whether they can carry out research end to end on the real, open Web remains insufficiently evaluated. The live Web is not a static retrieval setting. Its information is scattered across many sources, many modalities, and heterogeneous formats, and getting at it often means working through complex interaction mechanisms. An agent has to search, gain access, and integrate evidence on its own, while it is running. A benchmark built from a fixed collection of documents cannot test any of that. The pages do not move, nothing has to be clicked through or opened, and the evidence is already gathered in one place. What has been missing is an evaluation that puts the agent on the Web as it actually is.

The authors define a task for that situation, called OpenDS. An agent must explore the real Web by itself, find relevant evidence in several independent sources, and connect them, using Web search, browser interaction and evidence processing. At the end it has to arrive at a single factual answer that is unique. The measure is whether it gets there. Because the answer is a single verifiable fact, success can be scored plainly, while the path to it is left entirely to the agent: which searches to run, which pages to open, which files to process, and when it has gathered enough.

From that formulation comes OpenDS-World, a benchmark of two hundred challenging tasks. They span diverse information modalities, heterogeneous file formats, and tasks that require interaction just to reach the information. Each task was independently verified by humans, to make sure the evidence is valid and the answer is unique. Twelve frontier models were run under one shared tool environment and one execution protocol, so differences come from the models and not the setup.

The best model reaches thirty-nine percent accuracy. Humans, across one thousand independent attempts, reach twenty-one point one percent. These are hard tasks for people as well as for machines, and on this measure the leading model is ahead.

Beyond accuracy, the authors analyse tool usage, search cost, when agents decide to stop, and how they recover from failure. Those analyses show substantial differences in research strategy between models, and persistent difficulty in two areas: coordinating tools, and managing a search that runs over a long horizon.

OpenDS-World offers a single platform for assessing autonomous deep search where it actually happens, on the live open Web. The headline figure says the tasks are far from solved: the strongest model gets fewer than two in five right. The analyses say the shortfall is less about finding a page than about orchestrating many steps and tools over a long search. That points to where the work is: not in a single lookup, but in keeping a long investigation on track from the first query to the final answer.