RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

RoboJudge. Multimodal Language Models as Judges for Embodied Video Generation. By Siyuan Ma and ten others, from the Chinese University of Hong Kong, Westlake University, and four other institutions. Published on the twentieth of October. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.

Video generation models are promising foundations for embodied world models, with a condition attached. The rollouts they generate have to be physically valid, and they have to be faithful to the task instruction. Multimodal large language models are increasingly being used to judge exactly those properties. Whether their judgments agree with human judgments on embodied video, however, has not been systematically evaluated. That leaves a loop with an unchecked link: generators are scored by judges, and nobody has measured how closely those judges track what people see when they watch the same robot or hand perform the same task.

RoboJudge-Bench is built to evaluate MLLMs in that judging role. It measures two dimensions the authors treat as fundamental. Physical Adherence asks whether a video is physically valid. Instruction Alignment asks whether it is faithful to what it was told to do. The two are kept apart because a video can satisfy one and fail the other: it can obey physics while doing the wrong task, or perform the right task in a way that could not physically happen. A judge is assessed by how well its ratings on each dimension agree with the human ones.

The benchmark contains eight hundred videos annotated by humans. They come from eight source corpora and ten video generators, and they span three kinds of manipulator: grippers, human hands, and dexterous hands. Drawing on many corpora and many generators means a judge is not being tested on the habits of a single model or a single kind of footage.

Existing judges align with human ratings unevenly. General-purpose MLLMs agree less with people on physical adherence. Specialized video judges, designed for physical reasoning in general domains, do not consistently transfer to embodied scenarios. Neither the broad models nor the physics specialists, in other words, can simply be trusted to stand in for a human rater here.

Those findings motivate a training set. RoboJudge-Train comprises twelve thousand videos, annotated with twenty-five thousand human-verified scores and with rationales produced with model assistance, covering both Physical Adherence and Instruction Alignment. A nine-billion-parameter MLLM is fine-tuned on it, first with supervised learning and then with reinforcement learning, and the result is RoboJudge-9B. Among the model judges evaluated on the benchmark, it reaches the highest overall agreement: a Pearson correlation of zero point seven one nine. That is a statistically significant gain of zero point zero eight nine over Gemini-3.7-Flash, the best-performing proprietary baseline in the evaluation. Training on RoboJudge-Train also brings gains on external benchmarks, including VideoPhy2 and Physion-Eval.

The intended users are the people building video generators for embodied settings, who need a judge they can trust on physics and on instructions. The authors put RoboJudge-9B to that use themselves, as a downstream evaluator that benchmarks video generation models on Physical Adherence and Instruction Alignment.