MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances

MAMHOI. Factorizing Scene-Aware Human-Object Interaction through Affordances. By Mingyuan Lei and two others, from Nanyang Technological University. Published on the eighth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Generating a realistic interaction between a person and an object inside a complex 3D scene takes two abilities at once. One is reasoning about whether an interaction is feasible in that environment. The other is synthesizing convincing motion of the human and the object together. The difficulty is supervision. Human-scene datasets are rich in motion that respects the environment. Human-object datasets capture the detailed dynamics of an interaction. Data that pairs human, object and scene together is scarce. Each kind of data teaches one of the two abilities, and almost none teaches both at once.

MAMHOI answers with a factorization. Between scene understanding and motion synthesis it places an explicit interface built on motion affordances, so the problem splits into two halves that can each be learned from the data that suits it.

A scene-conditioned model goes first, predicting where and how an interaction can feasibly be carried out. An affordance-conditioned interaction model then takes that prediction and generates the matching human-object motion. Because the affordance sits between them, scene understanding and interaction dynamics are learned from complementary sources of supervision, and paired data covering human, object and scene is no longer required.

Experiments in complex indoor environments show that MAMHOI reduces penetration between the object and the scene, while better preserving the quality of the interaction between human and object. Those two measures pull in different directions: one asks whether the object respects the room, the other whether the person and the object still move together convincingly. Improving the first while keeping the second is the outcome reported, and the result is scene-aware interaction that is more realistic and more physically feasible.

The move here is to stop waiting for the dataset that does not exist. Two kinds of data already cover the two halves of the task, and a shared affordance interface is what lets each half be trained on its own kind.