Click3R: Interactive Stereo 3D Reconstruction with Sparse Correspondence Clicks

Click3R. Interactive Stereo 3D Reconstruction with Sparse Correspondence Clicks. By Hao Gu and two others, from the University of Virginia. Published on the ninth of November.

Feed-forward large 3D reconstruction models have made a great deal of progress, yet each one produces a single prediction, and that prediction is final. When such a model fails, there is no mechanism for correcting its mistakes. Some scenes are especially prone to failure: repetitive textures, symmetric structures, large changes of viewpoint, and regions with no texture at all. In each of these, the geometry is ambiguous, and the model has nothing to settle the ambiguity with.

Click3R turns reconstruction into an interactive process. Given an image pair, a user provides a small set of cross-view point correspondences: clicks that mark where the same point appears in both views. Those sparse clicks become geometric constraints that let the model resolve what it could not resolve on its own.

The constraints enter the 3D reconstruction networks through a lightweight point correspondence adapter, which is the channel through which a person's hints reach the model. To evaluate interactive reconstruction properly, the authors also curate a challenging benchmark, designed specifically to expose the failure modes of existing feed-forward 3D reconstruction methods.

In experiments, Click3R reduces reconstruction error with as few as a single correspondence click. Performance improves both on standard benchmarks and on the newly curated one, which was built to contain exactly the kinds of scene where feed-forward reconstruction tends to break. Gains on the standard sets show the correction is not confined to hand-picked hard cases. The abstract reports these improvements without giving specific error figures.

One click is a very small thing to ask of a person. Here it is enough to give a model a way out of a prediction it would otherwise be stuck with, turning a single irrevocable answer into one that can be nudged toward the right geometry by someone who can see what went wrong. The authors conclude that sparse human interaction is an effective and practical mechanism for correcting geometric reconstruction errors.