OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport

OTRetarget. Joint Robot and Object Motion Retargeting via Optimal Transport. By Guillaume Besset and seven others, from Inria, the École Normale Supérieure, and one other institution. Published on the twenty-ninth of September. No code repository is linked to it.

Transferring human motion to a humanoid robot means adapting what the person did to the robot's own morphology, while keeping the motion's interactions with the environment intact. Loco-manipulation makes this especially hard. The feet touch the ground and the hands touch the objects being moved, and those contacts must stay consistent even though the robot's body proportions are not the human's. A longer arm or a shorter leg changes where every contact would naturally fall.

Two things get in the way of the usual approach. Skeletal motion alone, the pose of the joints over time, does not fully describe those interactions with surfaces. And if the object trajectories are held fixed, copied straight from the demonstration, the motion can only adapt so far to a new embodiment. The robot is left to reach for an object that sits where the human's body needed it to be.

OTRetarget is a unified approach that retargets the robot's motion and the motion of multiple objects together, from human demonstrations. Instead of skeletal motion alone, it represents surface interactions with three quantities: signed distances, closest surface points, and relative directions. Entropic optimal transport then transfers those quantities across the human, robot, and object geometries. An interaction defined between a person and a box becomes a target defined between a robot and that box.

Those interaction targets are folded into a constrained inverse kinematics formulation. It balances two pulls against each other: preserving the contacts, and preserving the style of the motion. At every frame it optimizes the robot's pose and the objects' poses jointly, rather than solving for the robot around objects that are pinned in place.

Because the objects are part of the solve, the formulation accommodates both robot-object and object-object interactions. And it does so without rescaling the scene or the demonstration, so nothing has to be shrunk or stretched to fit the robot.

Validation is on OMOMO, with OmniRetarget as the comparison. On the robot-object interaction Jaccard score, OTRetarget reaches eighty-seven percent, where OmniRetarget reaches twenty-eight percent.

Depth error tells the same story from the other side: eight point seven millimetres for OTRetarget, against twenty-nine point three millimetres for OmniRetarget. Beyond simulation, the retargeted references are used to train whole-body policies with reinforcement learning, and those policies transfer to a physical G1 humanoid. The motions include a two-handed box pick-and-place onto a table.

Pinning the objects to their recorded paths was one of the limits on adaptation. OTRetarget's lever is to let them move with the robot inside the same optimization, steered by contact targets carried across bodies by optimal transport. A box set down on a real table is where that lever reaches hardware.