COSMI: COmpositional Synthesis of Multi-object Interactions

COSMI. COmpositional Synthesis of Multi-object Interactions. By Daniel Eskandar and two others, from the University of Tübingen, the Tübingen AI Center, and two other institutions. Published on the second of October. Its code repository is a placeholder: a readme, and nothing to run yet.

COSMI is a dataset of human-object interactions involving several objects at once. It holds two hundred and twenty-two thousand sequences, two hundred and seventy-five hours in all, with up to five objects in a sequence. The authors put that at nearly thirty times the largest multi-object capture. The gap it answers is a data gap. Generative models of human-object interaction are bounded by the data that exists, and while everyday activities involve several objects, most captured datasets record one at a time, because capturing several together is combinatorially expensive.

Each sequence is a multi-object activity assembled from single-object pieces, a person handling anywhere from one object up to five within the same clip. Taken together, that comes to more than two hundred and twenty-two thousand sequences, which is where the thirty-fold comparison with the largest multi-object capture comes from. The collection is not fixed in size. It can be extended by adding datasets, or even recordings of hands interacting with objects. And because it is built by composition, it grows combinatorially with the number of clips available rather than with recording time.

The construction rests on one observation: interactions are local, so single-object captures already contain the parts of multi-object activities. The authors compose them. Clips of single interactions are kept contact-consistent, mirrored to balance the hands, and transferred between bodies. Then two filters decide what survives. A language model and a set of geometric checks admit only the pairings that are plausible, semantically and physically. The language model speaks to whether a combination makes sense as an activity. The geometric checks speak to whether it can physically happen. A pairing has to pass both.

Its first user is the authors' own model, also named COSMI: a text-to-interaction diffusion transformer designed to follow how the data is built. Weight-shared object slots generate a variable number of objects, each predicted relative to the body parts that move it. On a benchmark with an unseen object and unseen interaction combinations, models trained on the dataset generalize to those combinations. COSMI outperforms baselines in text alignment and contact accuracy, and its margin is largest on the unseen object. Code, models and the dataset pipeline are to be released on the project page.

COSMI's wager is that the expensive thing never needed recording. If interactions really are local, the multi-object world is already present in single-object data, waiting to be recombined, and the limit on scale moves from hours of capture to the number of clips that can be paired.