HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

HandsOn. A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation. By Tianwei Luo and sixteen others, from Tsinghua University, the Beijing University of Posts and Telecommunications, and two other institutions. Published on the nineteenth of October. Its code repository is worked out: about two thousand source files, with a readme and a licence.

General-purpose manipulation requires reliable behavior across three kinds of variation at once: the embodiment doing the work, the objects being handled, and what the task demands. Measuring that calls for an evaluation that can vary all three, across different arms and different hands, and that can test whether a policy carries over to tasks it was never shown how to do.

HandsOn is a composable simulation platform and a benchmark for cross-embodiment dexterous manipulation. It supports five robotic arms and six dexterous hands, along with approximately two hundred rigid, articulated and deformable assets placed in randomized scenes. Qualitative rollouts on a real robot illustrate transfer from simulation to the physical world.

The benchmark pairs eighteen thousand curated human demonstrations with sixty-six human-verified tasks, in two suites. Core-50 supports demonstration-based learning and execution tests. Transfer-16 asks for more: asset and relation transfer, skill composition, and deformable manipulation, all without training demonstrations for the target task. One suite checks whether a policy can do what it was shown. The other checks whether it can go beyond that.

The methods evaluated span diffusion, vision-language-action, world-action, and prompted VLM-based approaches. The best trained multi-task policy reaches twenty-nine point one percent success on Core-50. Without any benchmark finetuning, GPT-6, using move_eef for palm-and-finger control, reaches five point two percent on Core-50 and one point nine percent on Transfer-16, over six hundred and sixty trials.

Stage diagnostics on the trained policies locate where things break. They fail at establishing contact, at coordinating actions, and at maintaining goals they had already achieved. Composed tasks add sequencing errors on top of those.

Read together, the figures describe an open frontier. Even the strongest trained policy succeeds on fewer than a third of the core tasks, and the diagnosed failures sit in physical interaction itself rather than in perception or planning alone. HandsOn is offered as a shared basis for investigating manipulation generalization and the limits of reliable physical interaction.