Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Haocheng Xi, Yiming Xie, Hexu Zhao, Yiwen Zhang, Michael Liu, Thomas Creavin, Kurt Keutzer, Xiuyu Li and 3 others
University of California, Berkeley · Impossible, Inc. · University of Texas at Austin
Published September 17, 2026 · on the feed September 18, 2026 · ▲ 50 · arXiv 2609.20744
The abstract
Video diffusion models repeatedly process long spatiotemporal token sequences during denoising, making attention a major computational bottleneck. Linear attention offers an appealing alternative and has been widely adopted in recent large language models, but directly applying it to video models often fails to preserve the fine-grained interactions required for high-quality generation. We present Video DeltaNet (VDN), which combines local Softmax attention with bidirectional linear memory for long-range video context. Its linear branch introduces Video Delta Attention (VDA), which updates memory once per frame by jointly incorporating its spatial tokens. Separate output projections and learnable gates calibrate the two branches, while a staged teacher-alignment recipe progressively introduces the new pathway into pretrained models. We instantiate VDN on MiniMax H3, applying the hybrid to video-to-video interactions while retaining Softmax for interactions involving text or audio. With eight-step distillation and an optimized SGLang serving stack, VDN-H3 completes DiT denoising for a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs, corresponding to a 14.5x speedup over the 50-step dense H3 baseline on the same GPU count.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
Video DeltaNet. A Video-Native Hybrid Attention for Livestream Video Generation. By Haocheng Xi and ten others, from the University of California, Berkeley, Impossible Inc, and one other institution. Published on the seventeenth of September. Its code repository is worked out: about ninety source files, with a readme and a licence.
Video diffusion models process long sequences of spatiotemporal tokens over and over during denoising, which makes attention a major computational bottleneck. Linear attention is an appealing alternative, and recent large language models have widely adopted it. Applied directly to video models, though, it often fails to keep the fine-grained interactions that high-quality generation needs. The saving is real, but so is the loss, and for video generation the fine detail is the product.
Video DeltaNet, or VDN, is a hybrid. It combines local Softmax attention with a bidirectional linear memory for long-range video context. The local attention keeps fine detail, and the linear memory carries context across the video cheaply.
The linear branch introduces Video Delta Attention, or VDA, which updates memory once per frame by folding in that frame's spatial tokens together. Separate output projections and learnable gates calibrate the two branches. A staged teacher-alignment recipe progressively introduces the new pathway into pretrained models, rather than training from nothing. The new branch is brought in by stages and aligned against the original model as a teacher, so the pretrained model's behaviour is the reference throughout. On MiniMax H3, the hybrid is applied to interactions between video tokens, while Softmax is kept for any interaction that involves text or audio.
With eight-step distillation and an optimized SGLang serving stack, VDN-H3 completes DiT denoising for a fourteen point three second video at seven hundred and sixty-eight p, in six point seven seconds, on eight NVIDIA B200 GPUs. That is a fourteen point five times speedup over the fifty-step dense H3 baseline on the same number of GPUs. The speedup combines three ingredients: the hybrid attention, distillation down to eight steps, and the optimized serving stack.
The title sets the aim at livestream video generation, and the design gets its speed by being selective about where attention stays exact. Softmax is kept within local neighbourhoods and wherever text or audio is involved, and the cheaper linear memory carries the long-range video context.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is attention, eating most of a video model's time.
You are speeding up a video generator. Attention eats more than 85% of the denoiser's runtime, because every token is compared with every other token and the cost grows quadratically with sequence length.
The road ahead
- 1 · The problem ~6 min Attention dominates the cost of video generation, and the cheaper compressed layer from language models loses the fine detail video needs.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Split attention by distance in time: exact Softmax for nearby frames, a cheap fixed-size memory for distant ones, updated frame by frame.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build the hybrid layer, align it to the pretrained teacher in stages, distil to eight steps, then serve it through optimized SGLang.
- 6 · The experiment ~6 min You’ll guess: Combining the hybrid attention, eight-step distillation and an optimized serving stack, how much faster was VDN-H3's denoising than the fifty-step dense H3 baseline on the same GPUs?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.