AutoCompress: Automated Hardware-Guided Model Compression

AutoCompress. Automated Hardware-Guided Model Compression. By Sheldon Liang and six others, from NVIDIA. Published on the fifth of November.

Pruning a large language model has to keep quality while meeting constraints on latency, throughput and memory that are specific to the hardware and the workload. Existing workflows mostly target a parameter count, which leaves practitioners choosing model sizes by hand and hoping they fit. Parameter count alone does not predict those metrics. They depend on the architecture, the workload, the hardware, and the serving engine.

AutoCompress builds the deployment constraints directly into differentiable architecture search. Instead of aiming at a size and checking speed afterwards, the search is steered by the quantities that actually matter in serving. Latency, throughput and memory limits stop being something checked after the fact and become part of what the search optimizes.

It jointly optimizes global structure and per-layer structure, so the resulting architectures can be heterogeneous: different layers can end up shaped differently, rather than every layer being cut by the same amount. The guidance comes from serving-engine cost models, which estimate how a candidate architecture will behave when served. Those cost models were validated against end-to-end measurements of latency, throughput and memory, so the search is steered by estimates that have been checked against real runs.

Three settings are tested: long-context local coding, batched datacenter serving, and latency-sensitive in-game interaction. Against Minitron, a state-of-the-art pruning framework, AutoCompress raises average accuracy across eight benchmarks by six point five seven percentage points on RTX 5070.

On RTX 5080 the gain is two point three five points, and on H100 it is zero point five seven points. Separately, AutoCompress shrinks Nemotron 3.5 Super from one hundred and twenty billion parameters to seventy-four billion parameters, so that it serves on a single B200 in BF16 at a context of one hundred and twenty-eight thousand tokens, keeping ninety-five percent of its mean benchmark score.

The shift is in the target. A parameter count is a proxy, and the work argues it is a poor one. Optimizing against modelled serving cost on the actual hardware lets the search land on shapes a uniform shrink would not produce.