Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs. By Noam Rotstein and seven others, from NVIDIA, the Technion, Israel Institute of Technology, and one other institution. Published on the twentieth of October. No code repository is linked to it.

Video generation models can now show remarkably realistic motion. What is unclear is how to use that motion once it leaves the pixels: how to animate a particular 3D asset with it, and replay the animation inside that asset's 3D scene. The authors' example makes the gap concrete. A video model can show a monkey jumping over a crossbar. It cannot hand back that jump as 3D motion you could replay, placed in the 3D scene where it happened.

Earlier models that combine video and motion fall into two camps. Some handle only human skeletons, and often add billions of parameters to do it. Others triangulate 3D motion from 2D pose maps that were themselves generated.

This work presents a model that co-generates the video and the 3D motion directly, for any rig it is given, together with a camera that registers the motion to the scene. The organising choice is separation. The 3D motion lives in its own model, one that supports arbitrary skeletons, and that model is connected to a pretrained video model through a small set of cross-attention layers. Video and motion are generated side by side, each in its own model, and the cross-attention layers are the narrow bridge that lets them inform one another. The motion model is not limited to human bodies, and the video model does not have to be rebuilt to accommodate it.

Keeping the two apart pays off in training. The motion model can first learn how skeletons move from motion-only data, which is easier to obtain and to augment than video paired with motion. The model also learns to locate each joint in the video frames. Knowing where the joints appear on screen is what allows the camera to be recovered, and with it, the 3D motion to be placed in the scene.

Training across many skeletons needs data for many skeletons. The authors use a generative video editor to place animated 3D assets into scenes that suit their actions, and they curate the edits so that the motion they already know stays valid. Combined with human and robot video, this yields nearly two hundred thousand training pairs. The editing step matters because the motion of each animated asset is known in advance: as long as the edit keeps that motion valid, every resulting video arrives with its 3D motion already attached.

In the evaluations, the model animates diverse rigs from text more faithfully than pipelines that generate the video first, and the motion it produces responds to the surrounding scene. Beyond generation from text, it runs in several directions. It turns video into rig motion. It turns rig motion into video. And it supports keyframe-based control of the 3D motion.

Picture the monkey again. Instead of a clip that ends at the screen, the model returns the jump as motion on the monkey's own rig, a camera that says where it was filmed from, and a place in the scene to replay it.