RoboJev: An Open Vision-Language Judge for Robot Success Detection

RoboJev. An Open Vision-Language Judge for Robot Success Detection. By Josef Liyanjun Chen, working independently. Published on the twenty-fourth of October. No code repository is linked to it, though it does publish its weights.

A great many robot manipulation videos carry no indication of whether the task was actually completed. The footage exists, but the label that says success or failure does not. Someone, or something, has to watch the attempt and decide. Doing that by hand across large collections of episodes is slow, so an automatic judge of success is useful wherever robot data piles up faster than people can label it.

RoboJev-27B is an open-weight vision-language model built to make that call. It takes a task instruction, ten frames of a manipulation attempt, and, when they are available, final frames from auxiliary cameras. From those it predicts whether the robot accomplished the task. The question is binary, and the design throughout is aimed at answering that one question cheaply and with a usable probability attached, rather than at describing the video in general.

The model is a variant of JEV-27B-VL. It was trained with a LoRA adapter of rank thirty-two, applied to the language model only, with the vision encoder kept frozen. The training set is fifty-two thousand five hundred and ninety-eight episode records.

Two further details shape its behaviour. A source-specific KL penalty applies only to research data, anchoring predictions on those sources closer to the original model. And the output is a calibrated two-token readout, which yields probabilities of success and of failure in a single forward pass.

On two hundred and sixty-eight held-out episodes from the same robot model, the SO-101, RoboJev-27B reaches ninety-five point two percent balanced accuracy, and an AUROC of zero point nine nine three.

Generalisation is tested two ways. On episodes collected by unseen labs or robots, four hundred and fifty-nine of them, the AUROC is zero point eight nine. For comparison, the base model given the same inputs reaches zero point nine on that subset. On clearly labelled episodes collected by the community, one hundred and thirty-five of them, accuracy is ninety-three point three percent.

On cost, inference takes a median of zero point two six seconds per episode, run serially on a single H100. The paper puts that at roughly zero point one nine per thousand success predictions, at a rate of two point six nine per GPU hour, not counting media processing or serving.

The release goes beyond the weights and inference code. A transparent evaluation comes with it: twenty thousand six hundred and twenty-eight predictions, organised into twenty-nine entries of model and protocol, so the results can be browsed and success detection can be benchmarked reproducibly. The paper also compares against strong API-based frontier VLMs. Those receive fewer input frames, four images from the primary camera only, and the author notes that the comparison is therefore not strictly like-for-like, because RoboJev-27B sees more frames at inference.

The pitch is a judge that anyone can run and audit. It answers one narrow question about a robot video, does so in a single forward pass, and arrives with its predictions published so that others can check the scoring for themselves. The caveats come stated alongside the numbers: on unseen labs and robots it sits below the base model's AUROC, and the frontier comparison is uneven by design.