From Short Clips to Real-Time Streams: Identity-Aware Supervision and Context-Efficient Generation for Video Face Swapping

From Short Clips to Real-Time Streams. Identity-Aware Supervision and Context-Efficient Generation for Video Face Swapping. By Zekai Luo and ten others, from Zhejiang University and the Zhejiang University of Technology. Published on the nineteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Video face swapping puts a reference identity onto a target video while keeping everything else about that video: its other attributes and its motion. Reference-guided flow-based methods have made progress, but three demands are still hard to meet at once. The identity has to transfer faithfully, the output has to be high fidelity, and generation has to run in real time over long video streams rather than short clips.

The authors' response is to rethink what is specific about this task, and they find two structural facts to exploit. The first is that a face swap changes only identity and leaves most of the visual content alone. Flow-matching supervision, though, is only weakly sensitive to changes of that kind. The second is that the target video already supplies the attributes that are not being edited, along with the temporal dynamics. The model does not need to generate those from scratch.

Each fact gets a design. For the first, a Joint Rec-ID Loss balances explicit identity supervision against latent reconstruction, with the balance set by the noise level. For the second, Face-RoI Generation and Short-Context Rollout Training let the model work with fewer spatial tokens and shorter temporal contexts, since the target video is carrying the rest. Put together, these form StreamingSwap, a reference-guided flow-matching framework for real-time streaming face swapping.

The work also introduces a dataset, HumanFaceVid. It holds a thousand hours of cinematic training data, plus evaluation sets of two hundred short videos and two hundred long ones.

In experiments, the method reaches competitive identity transfer and visual quality while running at twenty frames per second on a single NVIDIA GPU. It also stays stable over videos up to ten minutes long, which is the streaming claim the title points at.

Both halves of the design come from asking what this task leaves unchanged. Because so little of the frame should change, the loss is tilted toward the one thing that should. Because so much is already given by the target, the generator is handed less to produce. Speed and identity fidelity, which usually pull against each other, are each served by reading the task more closely.