The SIMPLE Law for Energy Cost of LLM Inference

The SIMPLE Law for Energy Cost of LLM Inference. By Hiari Pizzini Cavagna and four others, from the University of Bologna and CINECA. Published on the second of November.

Generative AI is billed per token, but tokens are not equally expensive. Every token enters the KV-cache and raises the cost of the ones that follow it, so energy grows non-linearly along a sequence even when its tokens look identical. No analytical model states what a token costs, and the most accurate existing model is a fit rather than a law: it absorbs architecture into fitted coefficients, so it has to be fitted again for every LLM. The claim here is that the cost can be written down as a law. The authors derive closed-form expressions for the computational and memory-access complexity of LLM inference, in terms of architectural hyperparameters and covering both MHA and GQA. Comparing a family of analytical models of increasing granularity, they arrive at SIMPLE, the Inference Model to Predict LLMs Energy. It is a four-parameter law. It separates prefill from decode, and splits decode further into KV-cache cost and next-token computation.

The law is tested on traces of thirteen dense LLMs from six families, running on NVIDIA H100 GPUs. With no per-LLM calibration, it reaches an R squared of zero point nine six nine and a mean absolute percentage error of eleven point nine seven percent. That error is three point one to three point five times lower than the state of the art.

Because its terms are analytical, the law explains as well as predicts. Prefill energy grows quadratically with the hidden dimension, so larger LLMs suffer most from long inputs. More aggressive GQA KV grouping cuts per-token energy by up to three times relative to standard MHA. Raising the reasoning token budget pushes output length past the analytically derived energy minimum, and energy then grows super-linearly. Accuracy also rises with the budget, but energy rises faster, so Intelligence per Joule peaks at an intermediate budget and falls by up to three point one four times across a four-fold range of budgets.

Agentic serving raises energy per request by thirteen times over non-agentic requests, because both the prefill term and the KV-cache term grow with input length. Prefix caching removes about ninety-three percent of the prefill cost, but the KV-cache term is unaffected, so even a cached agentic request costs four point eight times more.

The evidence is bounded by what was traced. The models are dense LLMs, thirteen of them across six families. The hardware is one GPU, the H100. The attention variants covered are MHA and GQA. The reasoning-budget and agentic results are readings of the same law on those traces, attributed to its prefill and KV-cache terms. Within that range, a four-parameter law with no per-model fitting tracks inference energy better than the fitted state of the art.

A token's price is flat, and its energy is not. SIMPLE gives a way to say, from architectural hyperparameters and without per-model fitting, where along a sequence that energy goes, and which choices, from grouping the cache to setting a reasoning budget, move it.

The SIMPLE Law for Energy Cost of LLM Inference. By Hiari Pizzini Cavagna and four others, from the University of Bologna and CINECA. Published on the second of November. This is the long version: the whole paper rather than the abstract.

Inference is billed per token, and the paper's opening complaint is that tokens are not equally expensive. Each token enters the key-value cache and raises the cost of every token after it, so energy grows non-linearly along a sequence whose tokens look identical. Inference accounts for up to ninety percent of a model's life-cycle energy, and a model with a lower listed price per token can cost up to twenty-eight times more in total than a nominally pricier one. Yet for a model not yet deployed, the only dependable way to learn its energy cost is to build it, serve it and measure it, by which point the architecture is no longer a choice.

Two ideas carry the rest. The first is the split between prefill and decode. Prefill processes the whole prompt in parallel and fills the cache. Decode then produces one token at a time, and every new token reads back every key and value stored so far, which makes decode memory-bound on long contexts. The second is how attention is organised. Standard multi-head attention keeps a key and value head for every query head. Grouped-query attention lets several query heads share one, shrinking the cache. The paper captures this in a single ratio, R: key-value heads divided by query heads. One means standard attention; a seventh means seven query heads share each cache head.

The best prior model, SweetSpot, from the same group, already derives decode energy in closed form and fits each model almost perfectly. But its coefficients are fitted one model at a time and silently absorb the architecture. Layers, hidden size and attention layout never appear as terms.

The derivation starts by counting. For every matrix multiplication in a decoder-only Transformer, in prefill and decode separately, the authors write down the floating-point operations and the memory reads and writes, with the key and value projections shrunk by the ratio R so that grouped-query attention is covered. Decode costs are summed over every generation step, where the cache holds the prompt plus everything generated so far.

That gives a long expression. The authors pruned it three ways: dropping terms with low statistical significance, discarding lower-order terms, and trying different groupings of what remained. Four fitted coefficients survive, giving three pieces of energy per generated token. Prefill amortization is the cost of reading the prompt, which scales with the square of the hidden dimension and linearly with layers and input length, spread across the output tokens. The cache term scales with hidden dimension, layers and the ratio R, and grows with input length plus half the output length, because the cache is still filling as generation proceeds. The next-token term is a fixed cost per generated token, linear in hidden dimension and layers.

Because the law is closed form, its minimum can be found by differentiating. Prefill amortization falls as outputs lengthen while the cache term rises, so energy per token has a sweet spot. That optimal output length grows with the square root of input length times hidden dimension, and shrinks as R grows.

The data are TensorRT-LLM traces from thirteen dense models across six families on H100 GPUs. The test closest to the thesis is leave-one-out validation, where a whole architecture is withheld during fitting. There the law reaches a goodness of fit of zero point nine four four and a mean error of about thirteen percent. A twenty-five-coefficient fine-grained model scores almost the same, so collapsing it to four costs essentially nothing, while the other model-agnostic baselines sit between thirty and forty-five percent error. Fitted per model, SweetSpot still wins, at about one point eight percent; the paper frames the gap as accuracy traded for generalisation.

The law then turns explanatory. Holding Llama 3.1 8B's other settings, moving from standard attention to a ratio of one seventh cuts energy per token by about three times. OPT 1.3B and Llama 3.2 1B share a hidden size, but OPT uses standard attention and more layers, and its predicted energy per token is more than three times higher, with the cache term rising from forty-eight to seventy-eight percent of the total. For a median GitHub Copilot agent request, with sixty-eight thousand context tokens, energy rises thirteen times over a short request. Prefix caching removes ninety-three percent of the prefill energy, yet the cached request still costs four point eight times more.

The paper names three limits. It covers only dense models on a single GPU: no mixture-of-experts, and only standard and grouped-query attention. It does not model batch size or concurrency. And the coefficients belong to one hardware and framework pair, so other accelerators, multi-GPU deployments and other inference stacks are untested.

The evidence has a further boundary. The reasoning analysis applies the law to Qwen3-1.7B and DeepSeek-R1-Distill-Llama-8B, neither of which is in the measured set, and borrows their accuracies from another paper. So the finding that Intelligence per Joule peaks at an intermediate reasoning budget, and falls by up to three point one four times away from it, pairs predicted energy with reported accuracy. The agentic figures are likewise estimates, not measured traces.

The idea to carry away is that a token's energy is not a constant but the sum of three parts with different shapes. Prefill cost shrinks as it is shared across more output. Cache cost grows with everything already in context. Next-token cost stays flat. Hidden size, layer count and the key-value head ratio set the size of each part, which is why model size alone fails to rank efficiency, and why long reasoning and long agentic contexts land where the cache term dominates.