VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis
Kangjie Chen, Xiangyu Li, Dongbin Zhang, Chaoda Zheng, Shijia Chen, Jinhao Deng, Hongbin Lin, Choo Sin Wai and 7 others
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University
Published September 27, 2026 · on the feed September 29, 2026 · ▲ 4 · arXiv 2609.33253
The abstract
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
The Brief
2:43A short podcast summary of the paper. Written and voiced by AI.
Transcript
VGGT-Diff. Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis. By Kangjie Chen and fourteen others, from XPeng Motors, the Chinese University of Hong Kong, and one other institution. Published on the twenty-seventh of September. Its code repository is worked out: about fifty-five source files, with a readme and a licence.
Novel view synthesis from only a few input views faces a basic trade-off. Methods built on reconstruction preserve the geometry they have observed, but struggle to fill in regions they have never seen. Methods built on diffusion bring strong generative priors, yet depend on correspondence between source and query views that is only implicit. One side is faithful but cannot imagine. The other can imagine but has no explicit hold on geometry.
VGGT-Diff is a multi-view diffusion model that routes geometry into generation. It takes visual geometry latents from VGGT-Ω and routes them into a pretrained video diffusion model, bridging the two regimes. The generative prior stays in charge of producing images, including the parts no input view covered. The geometry enters as conditioning, giving the diffusion model an explicit account of where things are instead of leaving it to infer correspondence on its own.
Each visual token is tied to a 3D point and a confidence. A confidence-aware Visual Geometry Router, or VGR, transforms those tokens into latent conditions aligned with the query view, preserving evidence about both front and back surfaces. The conditions guide joint denoising of the target views. Alongside that, Point-Track Residual Consistency, or PTRC, regularises the predicted-clean residuals along 3D tracks that are reliable, which improves stability across views. Robust geometry conditioning is added on top, combining regularisation during training with guidance at inference time.
Experiments show competitive or state-of-the-art performance, across both interpolation and extrapolation, and under different levels of viewpoint difficulty. No figures are given in the abstract, so the claim rests on that summary: the method holds up whether the requested views lie between the inputs or beyond them. The code is public.
Explicit geometry where it exists, generative prior where it does not. The router is where the two meet, carrying each token's point and confidence into the diffusion model so that denoising is guided by what was actually seen.
Paper Quest · 6 chapters · about 35 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A few photos, and a camera asked to go where none of them looked.
You have a handful of photos of a scene and must render it from new camera positions, some between the photos and some well beyond them.
The road ahead
- 1 · The problem ~7 min With only a few photos of a scene, you must render new views, and neither faithful reconstruction nor imaginative generation does the whole job alone.
- 2 · The key idea ~9 min Hand explicit geometry to a pretrained diffusion model as conditioning, aligned to each new view, and keep the generated views' errors consistent along shared 3D points.
- 3 · Choose your own research ~3 min
- 4 · Build the method ~8 min Get geometry tokens from VGGT-Ω, attach 3D points and confidences, route them into each query view with VGR, then denoise all targets together.
- 5 · What they found ~3 min
- 6 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.