Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

Sample Count Is Not Enough. Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling. By Mobina Kashaniyan and one other, from Iowa State University. Published on the sixteenth of September. No code repository is linked to it.

Test-time scaling can improve reasoning by generating several candidate responses and combining them. In sampling-based methods, the budget is usually described by N, the number of candidates. The claim here is that N is not enough. It says how many candidates are generated, not how. The same eight candidates can come from one batched generation call, or from several sequential calls with smaller batches, and those choices cost very different amounts. A budget stated as a count of candidates describes quantity, and says nothing about execution.

The study first confirms the accuracy side, on five hundred GSM8K prompts with Phi-3-mini and Qwen2.5-1.5B. As expected, raising N from one to eight improves accuracy by eight point four percentage points for Phi-3-mini, and by eighteen point four points for Qwen2.5-1.5B. Accuracy alone, though, shows nothing of the systems cost of that larger budget. That is the familiar half of the story, and it is what justifies spending more candidates. The rest of the study asks what spending them actually costs.

So N is fixed at eight, and four generation schedules are compared: one call of eight, two calls of four, four calls of two, and eight calls of one. Latency, throughput, GPU-hours, and gross GPU-device energy are all measured, with the total candidate count held constant. Holding the count constant is the key control. Any difference in cost then comes from how the eight candidates were scheduled, not from how many there were.

On A100 GPUs, eight serial calls use between four point six four and four point eight six times as much gross GPU-device energy as one batched call with eight candidates. Their P95 latency is between five point seven seven and six point one two times higher. The same pattern appears across three independently scheduled A100 nodes per model, and in short-output experiments on SciQ with V100 GPUs. Nothing about the candidates changes between schedules. The same eight responses cost roughly five times the energy, and around six times the tail latency, when they are generated one call at a time.

The conclusion comes with its conditions. Fewer calls with larger batches are more efficient when the candidates are independent, and when memory allows it. The accuracy measurements come from two small models on one benchmark, and the systems measurements from particular GPUs. The claim is not that batching is always possible, but that reporting N alone hides a large difference in cost.

The practical recommendation is about reporting. Evaluations of test-time scaling should state not only candidate count and accuracy, but also the generation schedule and GPU-level systems metrics. Two papers citing the same N may have spent many times different amounts of energy to get there. Without the schedule, a reader cannot tell a cheap result from an expensive one.