Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
Vida Adeli, Soroush Mehraban, Jacob Rommann, Harrison Sanborn, Cole Clifford, Babak Taati
Pickford AI · University of Toronto · Vector Institute
Published August 31, 2026 · on the feed September 10, 2026 · ▲ 20 · arXiv 2609.00369
The abstract
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-speech gesture diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent tokens, each depending only on the past. We then perform conditional diffusion directly in the causal latent space, conditioning on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.
The Brief
2:26A short podcast summary of the paper. Written and voiced by AI.
Transcript
Puppeteer. Object-Grounded Posture-Aware Co-Speech Gesture Generation. By Vida Adeli and five others, from Pickford A I, the University of Toronto, and one other institution. Published on the thirty-first of August. No code repository is linked to it.
When people talk, their hands and bodies move with the speech, and those movements also relate to the space around them: a posture held, an object pointed at. Generating such co-speech gestures is hard on three fronts at once. They must be coherent over time, aligned in meaning with the speech, and grounded in the surrounding objects. Earlier speech-driven gesture models concentrate on aligning gesture with audio. They do not explicitly account for posture constraints or for nearby objects, and so they miss the link between what a body does and the physical space it is doing it in.
Puppeteer is a gesture diffusion model that is both posture-aware and object-grounded, and it operates in a causal latent space. Causal here means each piece of the representation depends only on what came before it, never on what comes after.
Long gestures are first broken down into structured primitives. A causal variational autoencoder then encodes those primitives as latent tokens in temporal order, each token depending only on the past. Conditional diffusion runs directly in that latent space, conditioned on four things: the speech signal, the history of motion so far, a reference for the initial posture, and the geometry of objects in the scene. From those it synthesizes gestures that are physically consistent.
Because the latents are ordered in time, the formulation gives explicit temporal control, which in turn supports tasks such as gesture in-betweening and gesture completion.
Two contributions support the evaluation. The authors introduce new metrics tailored to co-speech gesture synthesis, to assess it beyond existing measures. And they built SceneGes, which they describe as the first curated synthetic 3D dataset of embodied co-speech gestures paired with the corresponding 3D objects, making object-grounded generation possible to train and test. In experiments, Puppeteer generates more diverse and more temporally synchronized gestures than prior methods, while also enabling gestures grounded in objects.
The name fits the design. A puppeteer's movements are shaped not only by the story being told but by the stage and what stands on it. Puppeteer conditions gesture on the speech, the body's starting pose and the objects nearby, building motion step by step in a space where the future never leaks into the past.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A speaker sits at a table and talks. The animation keeps putting hands through the wood.
You are animating a talking character. Its hands and body should move with the speech, in rhythm and in meaning, over long stretches.
The road ahead
- 1 · The problem ~6 min Speech-driven gesture models move a character's hands in time with speech, but they ignore its posture and the furniture around it.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Cut long gestures into short pieces, encode them as time-ordered latent tokens, and run diffusion there conditioned on speech, history, posture and objects.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Cut gestures into primitives, compress them into time-ordered latent tokens, then denoise those tokens conditioned on speech, posture and objects.
- 6 · The experiment ~6 min You’ll guess: Compared with prior gesture models, how did Puppeteer's gestures fare on diversity and on synchronization with speech?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.