CoDance: Learning Reactive and Compliant Human-Humanoid Interaction from Video

CoDance. Learning Reactive and Compliant Human-Humanoid Interaction from Video. By Zhuoqun Chen and two others, from Duke University. Published on the fourth of October. Its code repository is worked out: about four hundred source files, with a readme and a licence.

Partnered interaction between a human and a humanoid robot couples two hard things at once: locomotion, and continuous physical contact. The humanoid has to coordinate with how a person is moving, respond to the forces passing between them, and still keep its own movement stable and natural. A demonstration that only records poses says nothing about the forces, and the forces are much of the problem.

CoDance is a framework for learning that kind of reactive and compliant interaction from video. The test case is partnered dancing, chosen as a challenging instantiation: the humanoid must coordinate its footsteps with a moving partner while keeping continuous contact through both hands. The starting material is modest. A single video of two human dancers.

From that video, CoDance retargets the two dancers' motions into a robot reference and a moving partner. The key step is what the authors call a multi-link compliance augmentation. It transforms the kinematic demonstration into force-aware training data by adapting the robot reference under structured forces applied at both hands. In other words, the demonstration is turned into many variations of itself, each showing how the motion should give way under contact. Policies trained on this data are meant to do three things together: follow the observed partner, preserve the demonstrated locomotion style, and respond compliantly to physical interaction.

In simulation, the policies adapt their footsteps to changes in the partner, and they reproduce approximately eighty percent of the wrist displacement encoded by the augmented demonstrations. That figure measures how much of the intended give at the hands the policy actually shows. On a physical humanoid, CoDance enables sustained two-hand dancing with a human partner, including repeated transitions between moving forward and moving backward.

The thread running through the work is that compliance is not in the video, so it has to be put into the data. A recording of two dancers carries their poses and their footwork. By perturbing the robot's reference under forces at the hands, CoDance manufactures the missing part, and the policy learns style and give from the same source.