StreamErase: Towards Ultra-Fast Streaming Video Erasing

StreamErase. Towards Ultra-Fast Streaming Video Erasing. By Zhuobai Dong and six others, from Wuhan University, Alibaba Group, and two other institutions. Published on the fourteenth of October. Its code repository is worked out: about forty-five source files, with a readme.

Removing an object from a video is slow today. Current methods take minutes per video, and their processing is non-causal: they need to see the whole clip to work on any part of it. Between the latency and the non-causal design, real-time interaction is off the table. You cannot mark an object and watch it disappear as the video plays.

StreamErase aims at exactly that, with sub-second latency and streaming generation at up to twenty-six frames per second on a single GPU. The route there has two steps. First, a text-to-video model is augmented with a mask input, so it can generate high-quality videos in which the target objects are erased and coherent backgrounds are synthesized in their place. That model cannot run on the fly. So, second, its erasing ability is progressively transferred from this powerful bidirectional foundation model into an efficient unidirectional streaming eraser, one that can erase over long horizons without giving up visual fidelity.

Generating over long, possibly infinite horizons raises three challenges. The first is a domain gap: the model trains on finite lengths but must extrapolate to long ones. The second is quality: errors must not be allowed to accumulate. The third is speed: inference has to stay fast, without the computational cost rising as the context grows. Each of these gets worse the longer the video runs, and a stream, by definition, has no fixed end.

The answer is a carefully designed sliding-window causal attention with attention sinks. During training, the authors incorporate self-rollout with attention sinks, together with rolling the KV cache. That lets training simulate the extrapolation that happens at inference time, all within a fixed context window. Because the window stays fixed, generation runs at constant speed for videos of arbitrary length: a frame late in the stream costs what an early one did.

To evaluate this setting, the authors establish a dedicated benchmark called LongErase, for removing objects from long videos in a streaming fashion. On it, StreamErase achieves strong performance compared with existing video object erasing models, and it generalizes well to real-world videos. Among those models, it is the one that enables infinite-length streaming.

The experience this is built for is simple to describe. A user inputs a mask and watches the erasing results unfold in real time, which the authors describe as a truly interactive experience. Getting there meant two moves. The slow, high-quality bidirectional model became the source of the erasing capability rather than the thing that runs. And training was made to rehearse the long horizons that inference will face, so that a model trained on finite clips is not meeting an endless one for the first time when a user presses play.