ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference

ATEST. Adaptive Resource-Aware Test-Time Scaling for LLM Inference. By Mobina Kashaniyan and one other, from Iowa State University. Published on the eighteenth of October. No code repository is linked to it.

Test-time scaling improves the reasoning of a large language model by sampling several answers to the same question. The usual practice fixes how many samples to draw, and a fixed budget ignores two things. It ignores how hard the query is. And it ignores the cost of inference itself, which is not constant: under GPU contention, the energy a query consumes changes. The authors measure this directly. For the same four-sample query, energy rises by as much as four point one fold as the load on the GPU increases.

ATEST is a resource-aware test-time scaling framework that weighs both sides together. On one side is the expected value of drawing another sample. On the other is what that sample costs on the GPU right now. A query that is already settled should not keep drawing samples, and a sample that would be cheap on an idle machine may not justify its cost on a busy one. The framework comes in two modes, suited to different deployments.

The default mode, ATEST-Online, needs no offline calibration. It generates samples one at a time. As it goes, it estimates how stable the plurality answer is, monitors live GPU conditions, and learns the energy cost of each step online. From those signals it decides when to stop sampling. Nothing has to be measured ahead of time, because the cost model is built while the system is running.

The other mode, ATEST-Calibrated, instead picks a budget for each query in advance, using offline models of accuracy and cost. It favors deployments where the load is stable, since a budget fixed before the query runs cannot react to a machine that becomes busier while it does.

Both modes are evaluated on four reasoning and question-answering datasets, using Phi-3-mini on an NVIDIA A100 under controlled, varying load, with transfer tested on an NVIDIA L40S and on Qwen2.5-3B. Across the four A100 datasets, ATEST-Online answers every query without exceeding the energy budget. Its accuracy is the highest, or statistically tied for the highest, and it reduces how much it samples as the load climbs. That is the behaviour the design was after: spend freely when the GPU is quiet, and economize when it is contended.

On GSM8K at high load, ATEST-Online reaches sixty percent accuracy with no deferrals. ATEST-Calibrated reaches forty-four percent accuracy, and defers forty-two point four percent of queries. Ablations show that the online cost learning is what drives adaptation to load. Counting offline profiling, ATEST-Online uses forty-two times less GPU time and forty-eight times less energy than ATEST-Calibrated on the evaluated workload.

The authors are explicit about the price. Online control gives up some tail latency and throughput on short tasks in exchange for robustness when load varies, and offline calibration stays competitive when conditions are stable. The choice between the two modes comes down to how steady the machine underneath is expected to be.