ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing

ALIVE. Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing. By Zhenghong Zhou and three others, from the University of Rochester and Adobe Research. Published on the sixth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Today's video editors can insert an object into a scene. What they often struggle with is making that object take part in what is happening: being picked up, say, or manipulated. The object appears, but it does not interact with what is already in the video.

ALIVE is a framework for making inserted objects alive, in the sense of interacting coherently with the contents of the source video. The user supplies an edited first frame and an instruction that names only the object being added. Everything about how that object should behave comes from the model.

The key is the training data. The authors curate thirty-five thousand eight hundred editing pairs, mixing 3D-rendered, model-generated and real-world videos, together with general editing pairs from ROSE. Within each pair, the two videos differ in whether the target object is present, while the surrounding action is preserved. That contrast teaches an editor two things at once: coordinated object behavior, and preservation of the source.

Alongside the editor, a vision-language model is trained to predict interaction guidance from the same inputs, so guidance can be supplied without asking the user for more. For evaluation, the authors introduce the ALIVE-interaction benchmark. It assesses interaction fidelity, source preservation and visual coherence, using one unified protocol based on a vision-language model. They also evaluate on a general benchmark for video object insertion.

Even without the predicted guidance, ALIVE improves the Overall score over the strongest evaluated baseline by forty-three point nine percent on the interaction benchmark, and by four point four percent on the general one. Adding the guidance predicted by the vision-language model raises the ALIVE-interaction score by a further zero point nine five points, with no additional user input.

The point of ALIVE is that insertion is more than placement. By training on pairs where only the object's presence changes and the action stays put, the editor learns how an added object should join in, and the large margin on the interaction benchmark is where that shows most.