Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers

Sparsity Should Be ALLOTted by Output. Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers. By Tianze Xia and five others, from the University of Chinese Academy of Sciences, the Institute of Automation of the Chinese Academy of Sciences, and two other institutions. Published on the twenty-first of October. No code repository is linked to it.

In video Diffusion Transformers, full 3D attention is expensive. Block-sparse attention cuts the cost by computing only selected blocks of attention. That leaves a question of allocation: how much sparsity goes where. Existing methods either overlook the chance to optimise that allocation, or steer it with heuristic rules or local metrics.

Two observations set up the method. Sparsifying attention in different layers, and at different denoising steps, damages the fidelity of the final video to substantially different degrees. Yet which layer-step pairs are the most sensitive stays nearly the same from one input to the next. Sensitivity that is uneven but stable across inputs is something an allocation can be built on. That is ALLOT, short for Allocating Layer-step Latency by Output Tests, and it needs no training.

ALLOT treats each pair of a layer and a denoising step as a cell. It estimates how much each cell contributes to the error in the final video, using two procedures: cell probing, and trajectory calibration. It then optimises the densities of all the cells jointly, under a budget expressed in measured latency. Everything else is left alone. Existing selection strategies and kernels are kept, and only the allocation component is replaced. Two decisions are being separated here. Which blocks to compute within a cell stays with whatever selection method is already in use. How dense each cell is allowed to be is what ALLOT decides, guided by error in the final video rather than by a local signal.

At equal or shorter wall-clock time, ALLOT outperforms every compared allocation strategy when paired with XAttention, including a gain of eleven point eight decibels of PSNR on HunyuanVideo. Broader experiments add four more published pipelines, on Wan two point one, Wan two point two, and HunyuanVideo. Across all nine settings, ALLOT outperforms the default or ties with it.

The method's title states its premise. Sparsity should be allotted by what it does to the output, judged on the final video, rather than by a rule of thumb or a local measurement. The sensitive cells barely move between inputs, which is what makes that output-based judgment usable. And because ALLOT replaces only the allocation step, it works alongside existing selection strategies and kernels.