OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

OpenVLHarness. Real World Visual Agentic Tool-Use with Multimodal Memory. By Yu Zhou and ten others, from UCLA and Apple. Published on the nineteenth of October. Its code repository is worked out: about two hundred source files, with a readme and a licence.

Multimodal large language models, or MLLMs, have made rapid progress on visual reasoning. Yet many tasks need evidence that is difficult to obtain through internal reasoning alone. The authors ran a controlled study on twenty-two benchmarks, and simply increasing the inference budget of the base model yielded only modest gains. Thinking longer does not supply what is missing. Such tasks demand grounded perception, external knowledge, and numerical measurements, obtained through heterogeneous tools. Those tool outputs then have to be made interpretable to the model, and they have to be retained across reasoning steps rather than lost after one.

OpenVLHarness is a multimodal reasoning harness organized around two pieces: a capability layer, and persistent multimodal memory. The capability layer is about getting evidence in a form the model can use. The memory is about keeping that evidence available. A coding interface then lets the model compute over what has been kept.

The capability layer combines task-level tool interfaces with evidence rendering. It provides consistent access to heterogeneous backends, and converts their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data and web references as addressable artifacts throughout each query. The coding interface executes generated programs over that stored data, enabling computations that combine results from multiple tools. Together these support an iterative process of acquiring, inspecting, and composing multimodal evidence.

Evaluation covers twenty-two benchmarks with eight backbones, open and proprietary. Against matched baselines without tools, OpenVLHarness improves average scores by between nine point seven and twelve point six points. That includes eleven point two points with Qwen3-VL-8B, nine point seven with Kimi K3, and nine point nine with GPT-6 Astra.

A second comparison concerns cost. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms three alternatives running at maximum effort: the base model, Codex, and Visual Sketchpad. The margin is between seven point seven and eight point five points, while incurring at most thirty-seven percent of their model-token cost.

The opening study and the closing comparison are two sides of one argument. If extra inference budget brings only modest gains, what is missing is evidence from outside the model, and the harness exists to fetch that evidence, render it, and keep it. The medium-effort result shows the same thing from the other direction: better access to evidence outperformed a larger reasoning budget, at a fraction of the token cost.