RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On

RoboTryOn. Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On. By Seungyong Lee and three others, from KAIST and RLWRLD. Published on the third of November.

Robot foundation models perform remarkably well in robot control, but adapting one to a new task still leans heavily on teleoperated demonstrations. Egocentric human videos offer a scalable substitute, yet the usual ways of using them leave gaps. Overlaying rendered robots on human videos leaves a visual gap. Retargeting human poses into robot actions leaves an action gap. For robot embodiments with many degrees of freedom, both gaps are significant. A policy trained on such data either sees images that do not look like its own camera feed, or actions that do not match what its own body can do.

RoboTryOn borrows its framing from virtual try-on. Try-on keeps the person and swaps the garment. RoboTryOn keeps the demonstrated motion and swaps the human for the robot. Given a human video and a reference image of the target robot, it produces a video of that robot doing the same thing, while preserving the original task, the objects, and the background. The human video supplies everything about what happens. The reference image supplies only what the robot looks like. Everything else in the frame is meant to stay put.

Visual consistency is handled in two steps. Selected keyframes are translated first, autoregressively, and those keyframes then guide synthesis of the full video, with motion conditioned on the human hand skeleton. A video on its own is not yet training data for a policy, because it carries no action labels. For those, RoboTryOn uses an inverse dynamics model that incorporates depth features, inferring actions from the synthesized videos so that a policy can learn from them.

Experiments span multiple humanoid embodiments. On novel tasks, including challenging bimanual manipulation, policies trained with the synthetic data raise the average success rate from two percent to sixty percent, without any real robot demonstration of those tasks. A second comparison is about cost. Training with the synthetic data matches the performance of policies trained solely on teleoperated demonstrations, at approximately a third of the collection cost. The first result says the synthetic videos can stand in where no robot data exists. The second says that even where teleoperation is an option, the synthetic route reaches the same place for less.

The try-on analogy carries the whole design. What must survive the translation is the motion, the task, the objects and the scene. What gets replaced is only the body doing the work. Both results then point the same way: new tasks learned with no robot demonstrations of them, and parity with teleoperation for about a third of what collecting it costs.