Gen4P: Video Generators are 4D Perceivers

Gen4P. Video Generators are 4D Perceivers. By Yitong Xia and two others, from Nanyang Technological University and the University of Oxford. Published on the fifteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

The task is recovering 4D geometry and motion from a monocular video: the shape of a scene, and how it moves, from a single camera's view over time. Several jobs sit under that heading, from following points through 3D space to estimating depth across a video to recovering the pose of the camera. The question behind the title is whether a model pretrained to generate video already holds what all of those jobs need.

Gen4P is a unified framework built on the answer yes. A pretrained video generator, the authors show, can be repurposed into a feed-forward 4D perceiver, and it can do so without iterative denoising. That second part matters. The model is used for what it learned in pretraining, not for the step-by-step process it would normally run to make a video. The framework builds on on-demand querying: the model is asked for the 4D attributes that are needed, where they are needed.

Clean video latents pass through the encoder once, and a lightweight cross-attention decoder predicts whichever 4D attributes are requested. Querying on demand gives flexibility in where to sample spatially, and the proposed continuous-time queries add flexibility in time. To strengthen geometric knowledge, a pretrained video-depth LoRA is plugged in as a frozen geometry prior. It improves overall performance and adds no extra trainable parameters.

Gen4P achieves state-of-the-art or competitive performance on 3D point tracking, video depth, and camera pose estimation. Qualitative results on in-the-wild videos show it tracking complex motion while maintaining delicate structure. A single framework, then, is measured on all three tasks.

Nothing here is generated. The generator's pretraining is kept, its denoising loop is set aside, and what remains is read out on request, through queries that can land at any point in space and, with continuous time, at any moment in the video. The one addition, the depth prior, comes frozen. The authors take the results together as showing the potential of video generative pretraining for unified 4D perception.