MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

MC-Sparse. Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers. By Jiarui Chen and seven others, from Fudan University, Tencent HY, and five other institutions. Published on the fifth of October. Its code repository is worked out: about forty-five source files, with a readme and a licence.

Sparse attention is a primary way to reduce the latency of diffusion transformers on long-sequence tasks such as video generation and high-resolution 3D asset generation. But at high sparsity, existing methods can degrade the quality and fidelity of what they generate. Through controlled oracle comparisons, the authors trace that degradation to three sources. Token grouping imposes constraints. The selection of which interactions to compute is inaccurate. And the attention contributions of discarded tokens are simply lost.

Guided by that analysis, they propose Meta-Cached Sparse Attention, or MC-Sparse, a training-free framework. Its selection works at the level of individual key-value tokens rather than groups, while similar queries are organized into tile-aligned groups so the computation still runs efficiently on a GPU.

The name comes from what it caches. MC-Sparse stores metadata made of three parts: the query groups, the key-value indices chosen using exact attention probabilities, and the residuals between the dense and sparse attention outputs. That metadata is then reused across the denoising steps that follow. Each part lines up with one of the three diagnosed sources of loss. Selecting individual tokens answers the constraint that grouping imposed. Choosing indices from exact attention probabilities answers inaccurate selection. And keeping the residual between dense and sparse outputs answers the contributions that discarded tokens would otherwise take with them. Because all of this is cached and reused rather than recomputed at every step, the framework needs no training and adds little to each step's cost.

Across video and 3D generation models, MC-Sparse achieves higher fidelity to dense-attention outputs and larger denoising speedups than existing sparse-attention baselines, without visible quality degradation. Against dense attention, it delivers a one point eight times denoising speedup on Minimax-H3-Base. On 3D asset generation the speedup is two point three two times. Both come with negligible quality loss.

The paper starts by taking the gap between dense and sparse attention apart, measuring where the quality goes, and only then builds the method, one repair per leak. Caching the expensive decisions and reusing them across denoising steps is what keeps those repairs cheap enough to pay off.