InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

InteractiveWorld. A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants. By Qinsi Wang and nine others, from Meta and Duke University. Published on the twenty-fifth of October. Its code repository is worked out: about one hundred and seventy source files, with a readme and a licence.

The picture the authors start from is an old wish: an assistant that is always there, guiding you and reminding you of what you forget, whether that is a new dish you are cooking or the place you left your keys. Multimodal large language models are bringing that within reach in the form of egocentric AI assistants on wearable devices. They see what the user sees, and when necessary they give instructions for the user's physical work.

Testing that kind of help is the hard part. It usually requires real people, and real interaction data is both expensive and rare. So a basic question has stayed open: can today's MLLMs actually help people in the physical world? Without a way to run the assistant against someone who acts on its advice, there is no clean way to tell good guidance from guidance that only sounds good.

InteractiveWorld is built to answer it. The authors describe it as the first verifiable environment for closed-loop interaction between a human and an AI assistant. The difference from existing simulators is who does the acting. Those simulators are robot-centric. Here the assistant guides a human agent, and that human agent observes the scene, understands the instructions, and acts independently. The loop runs the way real collaboration would: the assistant instructs, the human acts, and the environment executes those actions and updates what both of them see next. What gets measured is whether the assistant's guidance actually leads to the right outcome in that loop.

On top of the environment sits InteractiveBench, described as the first dynamic benchmark for egocentric AI assistants. It holds one hundred and twenty multi-turn tasks across six categories. In every task the assistant guides the human in real time, and every instruction shapes how things turn out. Scoring is done with verifiable, state-based rewards, so success is read off the state of the world itself. Because each instruction changes what the human does and therefore what both of them see next, an early mistake carries forward through the rest of the task. A task cannot be passed by saying the right thing once. The assistant has to keep steering correctly, turn after turn, as the scene changes in response to its own advice.

Twelve frontier MLLMs were put through it, and the conclusion is that they are still far from reliable assistants. The categories that expose this most are the long-horizon ones. Even the strongest model, Claude Fable 5.1, reaches at most twenty percent on Guidance tasks. On Planning tasks its ceiling is twenty-five percent. Those are the best numbers in the field of models tested, and they leave most of the long tasks unfinished. The weaker models, by implication, do no better.

The authors present the work as a verifiable platform plus a rigorous evaluation set, a foundation for research on interactive MLLM assistants that genuinely help people in the physical world. What shifts is the framing. Instead of asking whether a model can act in a scene, the benchmark asks whether it can get someone else to act well, one instruction at a time, with the world keeping score.