Learning 3D Editing without Paired Supervision via Generative Prior Distillation

Learning 3D Editing without Paired Supervision via Generative Prior Distillation. By Hao Wen and six others, from Beihang University, the Central University of Finance and Economics, and two other institutions. Published on the fourth of September. Its code repository is worked out: about four hundred and fifty source files, with a readme and a licence.

Editing a 3D asset from an instruction is central to interactive content creation, and it runs into a hard bottleneck: high-quality paired training data, a 3D object before and after an edit, is severely scarce. Existing approaches try to get around this in one of two ways. Some rely on test-time optimization, which is slow. Others train on pseudo-pairs assembled through complex pipelines, and those pairs often introduce structural drift and geometric artifacts into what the model learns. Slow optimization makes editing impractical for interactive use, and flawed pseudo-pairs teach the model the flaws along with the edit.

This framework learns feed-forward 3D editing with no paired 3D supervision at all. The route is Generative Prior Distillation. Instead of ground-truth 3D pairs, the core idea is to distill visual, semantic, and geometric knowledge from powerful foundation models directly into a 3D editing model. The teachers already know what images look like, what instructions mean, and what realistic 3D assets are, and that knowledge becomes the supervision.

Supervision reaches the 3D representation through a differentiable rendering pipeline, as two complementary signals. At the main editing view, a 2D visual prior comes from an image editing model. At novel views, a semantic prior comes from a Vision-Language Model, there to enforce strict instruction following and to preserve the identity of the source object.

Supervising through 2D projections has known failure modes: geometric collapse, and inconsistency between views. To address those, the authors introduce a 3D-aware Distribution Matching regularization, which acts as a geometric prior. It operates in the 3D latent space, and it constrains the edited output to stay within the manifold of realistic 3D assets, a manifold defined by a pretrained image to 3D teacher model. The 2D signals say what the edit should look like and mean, and the 3D term keeps the result a plausible 3D object. Without that term, each view could satisfy its 2D teacher while the object as a whole fell apart.

Extensive experiments report superior instruction fidelity and cross-view consistency, with the method significantly outperforming state-of-the-art baselines. Those two measures line up with the two halves of the design: fidelity with the visual and semantic priors that read the instruction, consistency with the geometric prior that holds the views together.

Each foundation model contributes the kind of knowledge it holds, and the editing model is trained on their combined judgement instead of on examples of edits. Where paired 3D data is too scarce to learn from, what those models already know stands in as the supervision. The project code is available on GitHub.