OmniLatent: Unified Latent for 3D Part Modeling

OmniLatent. Unified Latent for 3D Part Modeling. By Yunhan Yang and eight others, from the University of Hong Kong, Tencent, and two other institutions. Published on the sixth of November.

Breaking a 3D object into parts is central to making assets people can edit. A chair you can pull apart into legs, seat and back is far more useful than a single sealed shell. Yet the usual workflow treats part decomposition as an afterthought. A mesh is completed first, and only then is it segmented, as a post-processing step. That ordering makes the result fragile. It inherits whatever geometric artifacts the mesh carries, it is confused by ambiguous texture, and it leans on part-level annotations, which are limited.

OmniLatent stops predicting part labels in isolation. It learns one shared latent space that jointly encodes four things: geometry, deformation, appearance, and binary part IDs. Putting them together is the point. Geometry and visual appearance are data-rich, part labels are data-scarce, and a shared space lets the abundant priors support the scarce part learning rather than leaving parts to be learned from their own thin supply.

The base model takes a single input image and, in one pass, generates geometry, deformation, texture, and surface part IDs. What comes out is already a part-aware 3D asset, with no separate segmentation stage at the end.

The pretrained model is then tuned for two extensions. The first is a point-prompt adapter. It maps arbitrary 3D point prompts, each carrying binary part IDs, into latent tokens, and that supports full segmentation, refinement, and subdividing a part further. The second is an amodal part completion model. Given partial surface masks, it completes them into a coherent decomposition of the full object. Three ingredients are named for that: dynamic per-part token allocation, cross-attention to the given geometry latent, and attention that alternates between a part-local view and a global one.

The reported experiments make three claims. OmniLatent produces part segmentations that are semantically meaningful and have precise boundaries. It supports interactive part editing. And it completes partial part observations while preserving the consistency of the object as a whole. The abstract describes these outcomes qualitatively and gives no figures for them.

The shift here is one of timing. Parts are not carved out of a finished mesh after the fact; they are generated alongside the shape, the deformation and the texture, from the same latent. Segmentation, editing and completion then become different ways of reading and prompting that one space, which is how a data-scarce task gets to borrow strength from the data-rich ones around it.