PhysInsert: Physically Plausible Object Insertion with Geometric Priors and Bidirectional Object-Scene Interactions

PhysInsert. Physically Plausible Object Insertion with Geometric Priors and Bidirectional Object-Scene Interactions. By Yue Gong and seven others, from Beihang University, MT Lab at Meitu, and one other institution. Published on the twenty-ninth of October.

Reference-based object insertion means taking an object from a reference image and placing it into a new scene. Doing that plausibly asks for more than copying appearance. The object's pose and lighting have to adapt to the scene, and the scene has to respond to the object. Existing methods often rely on masked inputs, generation-based pairs, or training pairs with limited viewpoint variation, which leaves pose and illumination adaptation, and the interactions running in both directions, insufficiently constrained.

PhysInsert jointly adapts the object and its surrounding scene. It starts from data: the Insert World dataset, built from strictly aligned Blender-rendered counterfactual pairs, with object references across viewpoints and lighting conditions. Because each pair is counterfactual and strictly aligned, the scene with the object and the scene without it differ only by the insertion, so the training signal covers what the object does to its surroundings as well as what the surroundings should do to the object.

Two components build on that data. Geometry-Guided Adaptation couples a VGGT geometry stream with image and text streams, under target-scene feature supervision, to guide spatially consistent insertion. Alongside it, 2D Object-Scene Spatial Supervision supervises both object occupancy and the environmental changes, and grounds spatial instructions in object regions through image-text attention. The authors also introduce PhysInsert-Bench, with dedicated cross-environment and cross-view cases that demand illumination and pose adaptation rather than appearance copying.

Experiments on PhysInsert-Bench and DreamBooth show improved image fidelity. A user study adds that placement is more accurate, and that lighting integration, perspective and pose plausibility all improve.

The thread through every part is that insertion runs both ways. Paired data that isolates the change, geometry to steer the pose, and supervision of how the scene itself changes all serve one aim: an object that looks like it belongs where it was put.