PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation

PAMI. Part Anchored Motion for Text to Human-Object Interaction Generation. By Chuqiao Li and four others, from the University of Tübingen and the Max Planck Institute for Informatics. Published on the twenty-ninth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Generating full-body human-object interaction from text means synthesizing two things at once, human motion and object trajectories, that match the text and stay precisely coordinated over time. Most methods represent the human and the object as separate trajectories, and predict the global coupling between them. That relationship is not fixed. It changes from moment to moment as contact begins and ends, and a single global coupling has to absorb all of that change without being told where it comes from. Learning that complex, dynamically changing relationship implicitly often goes wrong in three visible ways. The object drifts, contact is missed, or the two penetrate each other.

PAMI, a Part-Anchored Motion framework for Interaction generation, takes its key idea from the classic Hough Transform. Object motion is localized by letting body-part anchors vote for it. Instead of one global coupling, object motion is expressed relative to multiple body-part anchors. A model called PamiVAE learns an interaction latent space, and decodes frame-wise weights that aggregate these part-specific votes into one answer. The parallel with the Hough Transform is that no single anchor has to be right on its own. Each body part offers an estimate of where the object should be, and the weights decide, frame by frame, how much each estimate counts.

On top of that representation, interactions are generated in a coarse to fine hierarchy. PamiGen first generates a coarse human-object interaction from text, inside the structured latent space. PamiRefiner then recursively resolves the fine-grained contact geometry, using a hybrid surface-sensing representation. That representation combines two kinds of signal: long-range probes that capture the overall influence of each body part, and short-range sensors that resolve detailed contacts near the object's surface. The division of labour is clean. The first stage is responsible for getting the overall interaction right from the text, and the second for the detail where body and object meet.

Experiments on InterAct show PAMI generating more faithful interactions than previous methods, and more accurate object motion relative to the human. Its contact recall is fourteen point five percent higher than the previous state of the art. Extensive ablations validate both contributions: the part-anchored voting representation, and the hybrid surface-sensing refinement.

The change is one of reference frame. Asking where an object is, as a single global relationship to a whole moving body, is hard to learn implicitly. Asking where it is relative to each body part, and letting those local answers vote, turns one tangled relationship into many simpler ones, and a second stage then settles the contacts at close range.