Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion

Holo4D. Holistic 4D Reconstruction as Geometric Control for Video Diffusion. By Yushi Lan and five others, from the University of Oxford and Nanyang Technological University. Published on the eighteenth of October.

Reshooting a monocular video from a new camera trajectory asks for two things at once. The scene geometry and motion that were observed have to be preserved, faithfully, from the original footage. And the regions that only become visible from the new viewpoint have to be filled in, because the original camera never saw them. Two kinds of model each cover one half of that. Feed-forward 4D reconstructors recover the structure of what was observed. Video diffusion models supply a generative prior for completing what was not.

Holo4D joins the two halves. It is a video reshooting framework that takes reconstructed 3D trajectories and uses them as a holistic 4D scaffold of the source view, then lets that scaffold steer a pretrained video diffusion model. The reconstruction supplies the structure, and the diffusion model works inside it.

The trajectories are projected into the requested views, and their geometry and source appearance are encoded to guide the diffusion model. The scaffold does not act alone. Together with the source video and the target cameras, it grounds generation in what was actually reconstructed, while regions the reconstruction does not support are left to the generative prior to complete.

Experiments demonstrate the value of conditioning on trajectories, both for controlling the camera and for synthesizing novel views. The framework also accommodates different 4D reconstructors without retraining the video generator, so one trained generator can be paired with more than one way of recovering the source scene.

The division of labour is the design. Reconstruction says what is known, diffusion fills in what is not, and since the reconstructor can be swapped without retraining, future advances in source reconstruction should carry straight through to the reshot video.