RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

RealityBench. Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work. By Andrew Perrault and three others, from Relantic. Published on the ninth of October. No code repository is linked to it, though it does publish its weights.

Helping workers in the physical world through guidance has been a goal for AI agents for decades. One capability underneath it is understanding what the worker is doing, seen through the person's own camera or smart glasses, and helping them document it. Vision language models have advanced, but how well they handle this is uncertain, and hard to benchmark. Most physical work data is proprietary. And the work itself varies too much to study in a principled way. Kitchens and warehouses are structured, with predictable lighting, geometry and objects. Field tasks are not. Rooftop solar installation and vehicle repair are unconstrained and high variance.

The benchmark asks two questions of a model. The acceptability task asks whether a particular view would be suitable as proof of work. The documentation task asks whether an agent can understand the task-relevant state of the environment from that view. One is about whether the picture is good enough to file. The other is about whether the model actually reads what the picture shows.

To get around proprietary data and uncontrolled variability, the authors model real-world physical work in Blender, across eleven environments. Simulation means the variation can be built in deliberately. Each environment incorporates variants for both tasks, so that acceptability and documentation can be probed under controlled changes rather than whatever a collected dataset happens to contain.

Twenty-six vision-language models were evaluated. On documentation, the top models largely solve it when the answer is text visible in the frame. They are weak at reading measurements, and at judging small parts and spatial relations. The abstract's example is whether a tape measure leans.

Acceptability is harder to pin down, because the human raters disagree with each other. So the authors measure against a scorer that agrees with the raters' majority about as often as a single rater does. By that yardstick, the best models cannot be distinguished from a single non-author rater, and they do it at a cost within the range of a human pass.

Two further checks. In the three environments where acceptability errors could be traced, most wrong rejections came from misreading the scene, not from misreading the requirement. And on the three documentation questions for which real images exist, model rankings on rendered frames transfer to the real images.

That last result is what lets the rest stand. A simulated benchmark is only useful if it orders models the way reality would, and on the questions where the authors could check, it did. Within that frame the picture is uneven: reading printed text is close to done, while the small physical details a tradesperson would glance at, a measurement, a small part, a slight lean, are where models still fall short. And for the judgment call of whether a photo is good enough, the measuring stick is itself limited by how often people agree with each other.