HumanSkill-4D: Training-Free Language Querying of Animatable Gaussian Avatars in Space and Time

HumanSkill-4D. Training-Free Language Querying of Animatable Gaussian Avatars in Space and Time. By unsu Kim and three others, from the Ulsan National Institute of Science and Technology and NAVER AI Lab. Published on the seventeenth of October. No code repository is linked to it.

The task is new, and the authors name it avatar referring segmentation: given a language query, select regions of an animatable Gaussian avatar. A query can also describe a pose or a motion, and then the job is to select the time intervals where it happens. That would let a user edit an avatar locally, or inspect its motion, just by asking. Existing language Gaussian splatting methods run into two kinds of ambiguity here. Region ambiguity: neighbouring body parts and garments are hard to separate, and a query may refer to them only indirectly. Motion ambiguity: fine-grained motions can look alike in RGB.

HumanSkill-4D answers with a training-free framework. One multimodal large language model, an MLLM, and one set of geometric operators are shared across all avatars. What changes per avatar is memory. Rather than asking the language model to see the answer in pixels, the framework gives it named, structured things to reason over.

For space, each avatar gets a semantic memory that links body-part and garment categories to persistent Gaussian regions, along with descriptions of their appearance. The MLLM reads category names and descriptions and picks categories by appearance, by relation, or by function. For time, the joint trajectories that drive the avatar are stored in a motion memory, and the MLLM composes the geometric operators into programs that check the query's conditions against those trajectories, returning the matching intervals.

Two properties follow from building it this way. Because each selection comes from explicit category choices and operator outputs, every selection is traceable. And because operators can be composed or added, new kinds of motion condition can be supported as they come up.

To evaluate the new task, the authors introduce three human-annotated benchmarks. On them, HumanSkill-4D outperforms the baselines in part segmentation and in both three-dimensional and four-dimensional referring segmentation. It does so with compact memories and only minutes of setup per avatar.

Region ambiguity is handled by naming parts in a memory; motion ambiguity, by measuring trajectories instead of comparing appearances. Both moves replace something the model would have had to perceive with something it can look up or compute, which sits naturally with the two properties the framework claims: it needs no training, and every answer can be traced.