VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

VC-Attention. Value Smoothing and Softmax Casting for Low-bit Attention. By Xingyang Li and ten others, from Nunchux AI, MIT, and four other institutions. Published on the fourteenth of September. No code repository is linked to it.

Diffusion Transformers give state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant cost of deploying them. A low-bit attention kernel that can actually be deployed has to be both accurate and fast, and each goal has its own obstacle.

Accuracy is limited by outliers. A block's quantization scale is set by its largest entries, which leaves the typical entries squeezed into a narrow range of representable values. Earlier work smooths the queries and keys, but outliers in the values follow no fixed pattern across channels or across space and time, and they remain the main source of output error. Speed is limited by softmax. Low-bit Tensor Cores speed up only the two matrix multiplications, so the high-precision exponential between them becomes the longest stage of the pipeline on datacenter GPUs.

VC-Attention is a low-bit attention framework, needing no training, that tackles both obstacles at once. The V stands for value smoothing, aimed at the accuracy problem. The C stands for a fused probability cast, aimed at the speed problem. Both halves work inside the attention kernel itself, which is why nothing about the model has to be retrained.

V-Smooth reorders the value tokens using lightweight online clustering, so that the tokens sharing a hardware block quantize well together. It then quantizes only the residual left after subtracting the block's mean, and restores that mean afterwards from the row sum that online softmax already keeps. The correction costs almost nothing extra, because the quantity it needs is already being computed.

ExpCast-FP8 maps scores in the log domain directly to E4M3 probability codes with a single fused multiply-add. That removes both the FP32 exponential and the format conversion that would otherwise follow it. The expensive step is not made faster; it is replaced, and the probabilities come out already in the low-bit form the next multiplication wants.

VC-Attention is implemented for B200, B300, H200, RTX PRO 6000, and RTX 5090. It was tested on Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, and it improves fidelity over other low-bit baselines.

Against BF16 FlashAttention-4, the attention kernel runs one point four six to one point five nine times faster on datacenter Blackwell and Hopper, and two point three to three point six times faster on workstation cards. End to end, a whole clip is generated one point one three to one point one nine times faster in one setting, and one point three six to one point seven times faster in the other.

Each half of the name answers one half of the problem. Clustering and mean subtraction make the values easy to quantize, reusing a sum softmax already keeps. A single fused instruction removes the exponential that had become the slowest stage. Neither half requires retraining the video model it accelerates.