ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation

ReMoMask-2. Latent Retrieval-Augmented Masked Motion Generation. By Yiran Wang and three others, from the University of Sydney, Peking University, and one other institution. Published on the eighth of September. Its code repository is worked out: about sixty source files, with a readme.

Text-to-motion generation maps natural language onto human joint movements, with uses in gaming, VR and robotics. A retrieval-augmented version, RAG-T2M, does better on complex descriptions by conditioning on retrieved pairs of motion and text. Existing RAG-T2M models face two problems. Their retrieval and fusion are coarse-grained, and overlook the hierarchical, spatial-temporal topology of human motion. And there is a representation gap: retrieved evidence sits in a semantic space separate from the generator's own latents.

The work answers each problem in turn, with two systems that build on each other. For the first problem, ReMoMask is a structure-aware retrieval-augmented framework built from three components, each of which brings the structure of the body into a different stage: how motions are aligned with text, how retrieved motions are fused in, and how grounding is enforced. For the second problem, ReMoMask-2 moves retrieval itself into the generator's own latent space, so that what is retrieved is already in the form the generator uses.

ReMoMask's three components are these. Hierarchical Bidirectional Momentum contrastive learning, which aligns global and part-level features with text. Semantic Spatial-Temporal Attention, for fusion that is aware of topology. And Topology Structured Masking, which uses adaptive masking to force robust grounding at the level of parts.

ReMoMask-2 then rebuilds the retrieval database directly within the generator's pre-quantization latent space, and aligns text queries to it through a distilled lightweight projector. That lets the generator consume the retrieved motion's semantic content directly, without it having to cross from one space into another.

Evaluation runs on three datasets: HumanML3D, KIT-ML and SnapMoGen. Across them, the retriever reaches state-of-the-art accuracy. ReMoMask-2 attains the lowest FID on KIT-ML and on SnapMoGen. Notably, its single mask-transformer stage surpasses the full two-stage pipeline of the original ReMoMask, and delivers the fastest inference of the systems compared.

The second step is the one that simplifies. The first system adds structure to make retrieval and fusion finer-grained. The second removes a translation altogether. Once retrieved motions live in the same space the generator already works in, there is no representation gap left to bridge, and a single stage outperforms the two-stage pipeline it replaces while running faster.