MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations

MACaQ. Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations. By Lorenz K. Muller and five others, from Huawei Technologies. Published on the seventh of November.

Quantizing both weights and activations can cut the memory and compute cost of running a neural network. Recent methods for it, though, often need expensive optimization, and they propagate basis transforms across layers in ways specific to each architecture. Every new model family means new plumbing, and the whole network has to be considered together.

MACaQ is a post-training quantization method for MXFP4, with weights and activations both at four bits, and it treats every linear layer independently. The transforms that make quantization work are kept local to each layer and applied online, instead of being threaded through the network.

For each layer, MACaQ applies a per-channel scale and a block-Hadamard transform to the activations at run time, and absorbs the matching inverse transforms into that layer's weights. Because the transforms stay online, there are no cross-layer dependencies. That allows a simple streaming implementation of quantization, in which only one layer has to sit in memory at a time.

On top of this sits Flip-Refinement, meant to improve on round-to-nearest. It is a Hessian-aware coordinate-descent procedure that greedily moves individual FP4 codes to adjacent representable values.

Running the activation transforms at inference time has a cost, and it is small: two point three percent end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM. MACaQ reaches lower perplexity than DuQuant++ and MR-GPTQ. On Llama-3-8B it already matches their best perplexity using eleven times less quantization time.

Generality is shown by applying MACaQ, without architecture-specific changes, to an image diffusion model. Scale is shown by quantizing a model of one hundred and twenty-two billion parameters in sixty-four minutes, using only thirty-two point five gigabytes of GPU memory.

Keeping each transform inside its own layer costs a little at inference and buys a lot everywhere else. No cross-layer bookkeeping means no per-architecture rewiring, which is why a diffusion model can be quantized unchanged. One layer in memory at a time is what lets a model of that size fit on a modest budget.