WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents

WorldAuditBench. Interactive 3D World Auditing with Multimodal Agents. By Ziyan Jiang and seven others, from UC Santa Barbara, MIT CSAIL, and one other institution. Published on the thirtieth of September. Its code repository is worked out: about one thousand source files, with a readme.

Interactive 3D worlds are increasingly used to study intelligent behaviour, and that makes it important to find what is wrong with them efficiently. The anomalies are things like floating objects, walls you can walk through, or objects that do not fit the scene around them. Multimodal AI systems, both vision-language models and vision-language-action models, have shown potential for automating this kind of audit. But auditing a 3D world is hard in a particular way. It needs two distinct capabilities tightly coupled together.

One is action: navigating the world and searching for anomalies systematically and efficiently. The other is visual reasoning: understanding the environment and spotting anomalies from multimodal observations. Whether multimodal agents can actually couple the two, using visual reasoning to notice a possible anomaly and then taking actions to confirm it, remains largely unexplored.

WorldAuditBench measures exactly that coupling, through the task of world auditing. An agent is placed in a simulated 3D environment and has to find the anomalies in it, under a fixed exploration budget. It cannot audit what it never sees, and it cannot report what it sees but fails to understand. So the task asks for both halves at once: gathering evidence by moving, and interpreting that evidence by looking.

The benchmark has two hundred and thirteen anomaly tasks across thirteen environments, built with Unreal Engine five and with Three.js, and spanning five families of anomaly. Five frontier models are evaluated under two auditing paradigms. In the first, a VLA explores, and a VLM then identifies anomalies from what was gathered. In the second, a single end-to-end VLM agent lets its visual reasoning directly guide which actions it takes.

The two paradigms are two answers to the coupling question. The first splits the work, handing movement to one model and judgement to another, so the looking happens only after the exploring is done. The second keeps both in one loop, so that what the agent thinks it sees can decide where it goes next. Comparing them under the same budget is a way of asking whether that tighter coupling actually helps current models.

Across the evaluated models and both paradigms, success rates run from six point six percent at the low end to forty-two point three percent at the top. Humans reach eighty-three point four percent. Even the best configuration, in other words, lands at roughly half of human performance, and the results sit substantially below people across the board.

A world can look right in every frame and still have a wall you can walk through. Catching that requires an agent that looks, suspects, and then goes and checks. WorldAuditBench turns that loop into a testbed for how multimodal agents couple action with visual reasoning, and its first readings show current models still struggle to gather and interpret evidence while they explore.