VideoPhysEdit: Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction

VideoPhysEdit. Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction. By Conghan Yue and six others, from Fudan University. Published on the twenty-eighth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Video editing has advanced a great deal in recent years, and methods increasingly account for the visual consequences of an edit, such as changed shadows and occlusions. The physical consequences have been explored less: what an edit does to the motion that follows, and to the interactions it sets off. Nudge one object early in a clip, and everything it would later have touched should play out differently.

The authors formulate this as physical counterfactual video editing, or PCVE. Given a source video, a physical edit, and the frame at which the edit is executed, the goal is a counterfactual video showing the motion and interactions that result. The task is hard on several fronts. It requires understanding the physics of the scene and inferring the downstream motion and interactions an intervention induces. And there is neither paired factual and counterfactual data to learn from, nor dedicated metrics to evaluate with.

VideoPhysEdit is a new pipeline for this task in rigid-body scenes, and it needs no training. Its central move is to make the physical reasoning explicit, rather than hoping a video generator absorbs it. Being training-free also sidesteps the data problem just described: paired factual and counterfactual videos are scarce, and a pipeline that never learns the task end to end does not need them. The work is split between a part that reasons about physics and a part that renders video.

A novel physical scene reconstruction method recovers a scene that, when run under simulation, reproduces the motion and interactions observed in the source video. With that scene in hand, a physical edit becomes an intervention applied in simulation. The resulting trajectories then guide the generation of the counterfactual video, so the generator is told where things go rather than left to guess.

To fill the gaps in data and evaluation, the authors construct PCVE-RigidBench, a synthetic benchmark of paired source and counterfactual target videos with physical ground truth. Alongside it they introduce a metric, the Physical Edit Score. Physical ground truth is what makes the benchmark useful. Because the true counterfactual motion is known, an edited video can be checked against what should have happened, not just against whether it looks plausible.

VideoPhysEdit reaches substantially higher physical edit accuracy than both open-source methods and commercial models, while keeping visual fidelity competitive. Its Physical Edit Score is zero point three seven six, the only positive score among the methods compared. Every other method, open or commercial, lands at or below zero on the measure built for this task.

Real footage is covered qualitatively. Comparisons on real videos show that the pipeline applies to real-world scenes, and that it depicts the downstream motion and interactions induced by an edit better than the methods it is compared with.

Most editors ask what the changed frame should look like. VideoPhysEdit asks what the changed world would do next, and answers by handing that question to a simulation reconstructed from the video itself, then letting generation follow the trajectories that come back.