OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing

OmniXRef. A Unified Framework for Multimodal Multi-Reference Video Editing. By Xiaoyan Cong and nine others, from Brown University and Adobe Research. Published on the fifteenth of October. No code repository is linked to it.

Creative intent in video editing arrives in different forms. A text instruction, like remove the people. An image reference, for a subject's appearance. A video reference, for a camera movement. Or some combination of all three. An editor that accepts only one kind of reference at a time, or a fixed number of them, cannot follow intent expressed that way. Each kind of reference carries something the others cannot: words describe an operation, an image shows how something should look, and a clip shows how something should move.

OmniXRef is a unified framework for multimodal, multi-reference video editing. Users can combine variable numbers of text, image and video references to control diverse attributes of a source video, all within one model. The work comes in four parts: how the references go in, the model that uses them, the data it learns from, and a benchmark to test it.

It starts with a unified input design: a reference-bank interface, together with a positional scheme called R squared RoPE, introduced to help the model generalize to more references. The model itself, OmniXRef-Model, uses a single-stream diffusion transformer that jointly processes the source video and multimodal context from an MLLM, in one interleaved sequence. Training data for editing with video references is scarce, so the authors build OmniXRef-Data with synthesis pipelines for different editing tasks and combinations of references. Curriculum learning then raises task complexity and resolution step by step.

To evaluate video as a control modality systematically, there is also OmniXRef-Bench, a benchmark for transferring attributes from a reference video across five categories: camera, motion, content, style and visual effects. Comprehensive evaluations show OmniXRef-Model outperforming task-specific baselines and commercial models. One general model, in other words, beats systems built for a single kind of edit, as well as commercial tools.

Why build one model for all of this rather than a tool per task? Because the intent behind an edit rarely comes in a single form. The bank of references, the shared sequence and the synthetic data are all in service of letting text, images and video sit side by side in one request, with video treated as a control signal in its own right.