RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild

RoboColosseum. Evaluating Physical Intelligence at Scale In-The-Wild. By Tan Sang Nguyen and twenty-six others, from the National University of Singapore, the University of Washington, and seven other institutions. Published on the second of November.

A general-purpose robot policy has to work across robots, across tasks, and across environments. Evaluating one means running it in all that variety. The trouble is that the same variation that makes a physical trial informative also makes it hard to reproduce. A different room, a different robot, a different operator, and the result may not repeat. Testing broadly and testing reproducibly pull against each other.

RoboColosseum is a distributed evaluation framework built to hold both. It combines unified policy serving with standardized protocols, running across a network of locally operated robots in diverse environments. The variety comes from the network; the comparability comes from the shared serving and protocols. It measures through two tracks. The Open Track compares models with no task-specific adaptation, in blinded pairwise trials under matched local conditions. The Fine-tune Track measures how well vision-language-action policies adapt to twenty tasks across all four embodiments, with every policy given the same one hundred demonstrations per task. Partial progress is measured too, not only whether a task succeeds.

The initial study covers five vision-language-action policies and three frontier AI agents, so specialized robot models and general agents meet on the same hardware. The four embodiments are SO-101, Franka, Bimanual-YAM, and the G1 Humanoid, four quite different bodies for the same policies to run on. The Open Track alone gathers five thousand six hundred and ninety-nine pairwise comparisons, at one hundred and twenty-four sites. That is the scale in the title: many trials, spread over many places.

In the Open Track, frontier agents rival specialized VLA policies, while every system struggles to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101, and pi0.5 leads on Franka and on Bimanual-YAM. So the leader depends on the robot. Policy rankings agree only weakly across the task suites of different embodiments, with a Kendall W of zero point two four. Open Track rankings are also a poor guide to relative performance after fine-tuning, with substantial rank reversals on all three embodiments that the two tracks share.

Partial progress exposes differences that success alone misses. On the G1 Humanoid, no policy completes any of its five tasks, yet average progress ranges from thirteen point three percent to twenty-three point two percent across policies. A success-only score would call them all equal at zero. On cost, Open Track rankings stabilize as trials accumulate, and active matchmaking reaches comparable ranking agreement with substantially fewer physical trials than uniform sampling.

The framework keeps three questions apart: where robot policies generalize, how well they adapt to new tasks, and how far they get on tasks they fail. That the answers diverge, with Open Track rankings poorly predicting fine-tuned ones and a humanoid where nothing succeeds but progress still varies, is the case for measuring them separately, and for doing it across many sites at once.