RoboValue: A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models

RoboValue. A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models. By Shengbang Liu and eighteen others, from Tsinghua University, AgiBot, and four other institutions. Published on the nineteenth of October. Its code repository is worked out: about one hundred source files, with a readme and a licence.

A robotic value model looks at visual observations and an instruction, and judges how a task is being carried out. That judgment is used as feedback, both for learning policies and for monitoring execution. Existing evaluations, though, cover only selected aspects of understanding an execution. The reliability of the feedback itself has not been tested thoroughly enough, which matters when that feedback is what a policy learns from.

RoboValue is a fine-grained benchmark, in simulation and in the real world, that evaluates value models in one unified framework. Through shared interfaces it measures four complementary capabilities. Task-state understanding: knowing where the task currently stands. Temporal progress monitoring: following how that changes over the course of execution. Reasoning about failure and recovery: recognising when things go wrong and when they are put right. And consistency of the values themselves. Taken together, the four ask not only whether a model can tell success from failure, but whether its feedback holds up across everything that happens in between.

Diagnostic trajectories and variations of the instruction probe whether a value judgment really reflects what the task requires, and whether it tracks changes as execution unfolds.

The accompanying RoboValue-Dataset covers fifteen manipulation tasks in simulation and twenty in the real world. It supplies expert demonstrations for training, and annotated trajectories for fine-grained evaluation. Representative value models are tested in zero-shot and one-shot settings, first under standard conditions, and then under separate shifts in robot embodiment and in environment.

The results expose errors in three places: grounding fine-grained instructions, tracking progress in a way that depends on history, and assessing recovery. These errors appear even in models that score well on outcomes and on forward progress. A good headline score, in other words, does not certify the finer judgments.

There is also an inconsistency. The same semantic subtask can receive substantially different value gains depending on which valid order the execution followed. Both orders complete the task correctly, so a value model that scores them differently is giving feedback that depends on something other than progress.

A value model is only as useful as the feedback it gives, and RoboValue's results locate where that feedback goes wrong. The authors frame them as concrete targets for value supervision. The dataset and a public leaderboard are there to keep measuring how reliable that feedback becomes.