WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites

WebFovea. When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites. By Jiangang Han, working independently. Published on the seventh of October. Its code repository is public and real: about thirty-five source files.

WebFovea is a vision-based web agent that placed second in the WebRetriever Challenge 2026, with a final score of fifty-seven out of a hundred. The challenge tests agents end to end on Protocol III of the WebRetriever benchmark. The agent starts from an entry address on a live website, must operate that site's own interface, and has to come back with an answer that can be verified. A capable multimodal large language model is necessary for this. The author's point is that it is not sufficient.

The model never touches the browser directly. Its decisions travel through the harness, which is the code sitting between the model and the page. At every step, four things have to go right. The model's reply must be parsed into the action it intended. That action must actually take effect on the page. The result must be reported back accurately. And the model must be shown the information it needs for the next step.

The central observation is about where things broke. On real websites, many of the failures the author saw happened at one of those four stages, not in the model's reasoning. The model was right, and the click was wrong. That shifts where the engineering effort belongs. A stronger reasoner cannot fix a reply that was parsed into the wrong action, or an action that silently did nothing, or a result that was reported back inaccurately. The contribution, then, is a harness that treats each stage of the round trip as something to be made reliable in its own right.

The failures were concrete. A mismatch between coordinate spaces put every click at three quarters of its intended coordinates. Actions on native dropdowns, inside iframes, and in text boxes failed silently. None of these is a mistake of reasoning. Each is the harness letting a correct decision go astray somewhere between the model and the page. And chat-template tokens that the model generated itself contaminated four point nine percent of task episodes.

WebFovea hardens each of the four stages against failures like these. Around the loop it adds guardrails that keep the agent inside the challenge rules and inside its budget. The four-stage view itself does not depend on which model is used, though some of the individual fixes do.

The cleanest piece of evidence comes from the submissions. The same model was used in all four of them, and the official hidden-set score rose from thirty-one to fifty-seven. Because the model never changed, that rise reflects changes to the harness, up to the run-to-run variance that live sites introduce. The report covers the design, the evidence behind each component, negative results included, a failure analysis, and the limitations.

The title carries the argument: when the model is right but the click is wrong, the fault sits outside the model's reasoning. The fix is to make each leg of the trip from model to page and back reliable. The roadmap points one step further, toward routing different steps to different models, which fits a view in which the model is one part of the system rather than the whole of it.