Streaming Video Editing with Easy Adaptation
Yujia Hu, Jiajun Li, Zihao He, Songhua Liu
Shanghai Jiao Tong University
Published September 21, 2026 · on the feed September 22, 2026 · ▲ 10 · arXiv 2609.24788
The abstract
In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU 17 without any auxiliary acceleration techniques. Codes are available at https://github.com/YujiaHu1109/SVEET.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
Streaming Video Editing with Easy Adaptation. By Yujia Hu and three others, from Shanghai Jiao Tong University. Published on the twenty-first of September. Its code repository is worked out: about five hundred and fifty source files, with a readme and a licence.
Pretrained video diffusion models are typically bidirectional: they see the whole clip at once. Streaming video editing needs the opposite, an auto-regressive process that edits as frames arrive. The question this work takes on is how to turn a pretrained bidirectional model into a streaming editor that still produces high-quality edits, with only light training on top of what already exists.
The authors start by systematically revisiting existing video-to-video diffusion approaches. From that review they draw two principles for adapting to streaming. The first is backbone feature disentanglement. The second is conditional frame independence. SVEET is the framework built on those principles, and with it comes a new paradigm for controllable video generation.
At the core is an auxiliary model branch. It encodes the source video with self-attention that is temporally independent, and its intermediate features are injected into the matching blocks of the backbone. That gives control that is compatible with streaming.
There is still a gap to close. The feature spaces of bidirectional and streaming models differ. To bridge them, SVEET uses a decoupled training scheme that explicitly enforces orthogonality between two optimization directions: one for video controllability, and one for model causality. Keeping the two apart makes the objectives compatible at inference, and it enables smooth zero-shot transfer of knowledge across backbone architectures that differ from one another.
In extensive experiments, SVEET achieves superior editing quality while keeping real-time performance. It reaches fifteen frames per second on a single H100 GPU, without any auxiliary acceleration techniques. That is the combination the paper set out to reach: high-quality editing, built on a pretrained bidirectional model, delivered in the auto-regressive form that streaming requires, and fast enough to keep up in real time on a single GPU.
Separation, at two levels, is how the design works. In the architecture, the source video is encoded frame by frame in its own branch, so conditioning never depends on frames yet to come. In training, the push toward control and the push toward causality are kept orthogonal, so one does not undo the other. Those two separations are what let a model trained to look at a whole clip edit one arriving frame at a time.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a great video model that insists on seeing the whole clip.
You want to restyle or inpaint a live video as frames arrive. Good pretrained video diffusion models exist, but they are bidirectional: they look at the whole clip at once, which a stream cannot offer.
The road ahead
- 1 · The problem ~7 min Good video editors see the whole clip at once, but a live stream only gives you frames as they arrive, and training a streaming editor is costly.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Learn editing in a side branch on the full-clip model, encode each source frame on its own, and keep training away from the full-clip-to-streaming shift.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build a per-frame control branch, train it on the full-clip model with an orthogonal constraint, then plug it into the frozen streaming model.
- 6 · The experiment ~6 min You’ll guess: Without any extra acceleration tricks, how many frames per second does SVEET reach for streaming editing on a single H100 GPU?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.