Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

Interactive Visual Gym. Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds. By Chenhao Zheng and thirteen others, from the University of Washington and Salesforce AI Research. Published on the fifth of November.

Picture an assistant that watches you cook and talks you through it. Every reply it gives can change what you do next, and so change how the task turns out. Visual interaction models of this kind, supporting users during ongoing physical tasks, need to be evaluated in a closed loop for exactly that reason. Existing benchmarks mostly rely on offline replay, where the recorded user never reacts to the assistant, so these effects cannot show up. A replayed recording cannot tell you whether good advice would have saved the dish, or whether a slow answer arrived after the mistake was already made.

Interactive Visual Gym is a framework for closed-loop evaluation, and also for training, of visual interaction models. It simulates two things together: an executable task environment, and the human user acting inside it. What gets measured is whether tasks are completed within a time limit, and the quality of the interaction, including factual grounding, actionable guidance and whether help arrives in time once response latency is counted.

The authors contribute three simulators that recreate everyday tasks with human-aligned action spaces, covering cooking, household manipulation and mobile interfaces. On top sits a user simulation harness conditioned on a persona. The simulated user collaborates with the assistant, but also makes controlled execution errors and other moves that depart from the plan. That matters, because an assistant that only ever meets a perfectly obedient user is never tested on the moments when help is needed most. The framework is instantiated as IVG-Bench, with three hundred and fifteen interaction scenarios.

Current vision language models, used as visual interaction models, show substantial limitations. The best performer, Gemini 4 Argon, completes only fifty-three point four percent of tasks within the time limit. It shows significant weaknesses in factual grounding and in actionable guidance, and once latency is taken into account it is often too slow to prevent errors.

A human study supports the protocol, finding seventy-eight point five percent agreement between the interaction-quality metric and what participants preferred. The environments also give a useful learning signal. In a proof-of-concept GRPO experiment in the cooking environment, closed-loop training improves both task success and interaction quality for the Qwen3.5-4B model.

Once the user can react, an assistant is judged not only on what it says but on when it says it and what happens next. Even the strongest model tested finishes barely more than half the tasks in time. Here the loop that exposes those weaknesses doubles as a place to start training them away.