SO-101 Bench: Accessible Spatial Manipulation

SO-101 Bench. Accessible Spatial Manipulation. By Truman Hickok, working independently. Published on the twenty-seventh of October.

Low-cost arms like the SO-101 have become the most common way into robot learning. Yet two things are missing for them. First, no released policy performs spatially constrained, language-conditioned tasks on these arms out of the box: you cannot download a policy, give it an instruction about where something should go, and expect it to work. Second, the existing SO-101 benchmarks run only on physical hardware, with simple tasks and few held-out objects, so they say little about how a policy copes with things it has not seen.

SO-101 Bench is a simulated benchmark for language-conditioned spatial manipulation. It has four tasks, scored as five families. The evaluation set is nine hundred and sixteen episodes, and within it are subsets aimed at particular skills: held-out appearances, rotate-to-fit planning, and memory under occlusion.

It is built on an Isaac Lab digital twin of a real SO-101 setup. There are one hundred and thirty-eight objects, divided into three splits: seen, unseen-instance, and unseen-class. Every learned policy is fine-tuned on the same three thousand six hundred and twenty-nine simulated demonstrations, under an equal compute budget, so differences between policies are not differences in how much they were trained.

A simulator is only useful if it ranks policies the way the hardware does, and the twin is checked on exactly that. Five policies trained without real data, four VLAs and one LLM agent, were run with the same weights on the physical arm, in an unseen environment with unseen objects. They kept their exact ranking: a Spearman rho of one, and an MMRV of zero. Simulation does overestimate success, but it orders the policies as the real arm does.

Across nine VLAs and world-action models, the best learned policies score near sixty-one percent. Three areas remain open: spatial placement, unseen object classes, and rotate-to-fit planning.

A different kind of system does better. An LLM agent that writes code and hands grasping off to a learned policy scores eighty-three point two percent. An ablation traces that lead to two choices, the choice of harness and the choice of language model.

What the benchmark offers is a simulator whose rankings match the hardware, for the arm that has become the common entry point. Its success rates run higher than the real arm's, but the order it puts policies in is the order the real arm puts them in, and that order is what a comparison needs.

SO-101 Bench. Accessible Spatial Manipulation. By Truman Hickok, working independently. Published on the twenty-seventh of October. Its code repository is worked out: about two hundred source files, with a readme.

Low-cost arms such as the SO-101 have become the most common entry point to robot learning. Yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box. And the existing SO-101 benchmarks run only on physical hardware, with simple tasks and few held-out objects. Two gaps sit side by side, then. The arm most people start on has no policy that handles spatial instructions without further work, and the ways of measuring progress on it are tied to a physical setup, kept simple, and thin on objects the policy has not seen.

SO-101 Bench is a simulated benchmark for language-conditioned spatial manipulation. It is built on an Isaac Lab digital twin of a real SO-101 setup. It has four tasks, scored as five families. Its objects number one hundred and thirty-eight, divided into seen, unseen-instance and unseen-class splits.

There are nine hundred and sixteen evaluation episodes. Subsets of them probe held-out appearances, rotate-to-fit planning, and occlusion memory. The object splits and these subsets answer the earlier complaint directly: there are held-out objects at two levels of novelty, and the tasks ask for more than simple placement.

For training, the benchmark supplies three thousand six hundred and twenty-nine simulated demonstrations. Every learned policy is fine-tuned on them under an equal compute budget, so differences between policies are not differences in training spend.

The key check is whether the twin ranks policies the way the real robot does. Five policies trained without real data, four VLAs and an LLM agent, were run with the same weights on the physical arm, in an unseen environment with unseen objects. Their ranking was exactly preserved. Spearman rho was one point zero zero, and MMRV was zero. Simulation does overestimate success, though: the order holds, the absolute numbers do not.

Across nine VLAs and world-action models, the best learned policies score near sixty-one percent. Three things remain open: spatial placement, unseen object classes, and rotate-to-fit planning.

An LLM agent that writes code and hands grasping to a learned policy does better, scoring eighty-three point two percent. An ablation traces that lead to two choices, the harness and the language model. So the strongest result on the benchmark comes not from a better learned policy alone, but from how the agent around it is assembled and which model does the writing.

For a cheap and widely used arm, the benchmark offers something that can be trusted for comparison even where it cannot be trusted for magnitude. A ranking that survives the move to real hardware means a result in the twin says which policy is better, while the open categories say where all of them still fall short.