Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models

Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models. By Mohammed Nayeem Khan and two others, from Intel Corporation. Published on the tenth of November.

Small local language models have two cheap levers. They can store weights in fewer bits to save memory. And they can average over a few reorderings of the answer options, to become less sensitive to the order those options come in. The question asked is whether the first lever makes the second one's problem worse, and which is the cheaper way to spend a fixed budget. The answer arrives as a decision rule: use six-bit weights, and spend the rest of the budget on up to four option orders, one pass each.

The models are Qwen3-0.6B and Qwen3-1.7B. Weight quantization is simulated by rounding to nearest, at eight, six, four and three bits, against an fp16 reference. Answers are read from the logits of the letters A to D, on ARC-Easy, ARC-Challenge, MMLU and HellaSwag, with three hundred items each. Six bits lose no accuracy against fp16. They actually score higher, by one point eight points for the smaller model and zero point nine for the larger, a gain the authors do not explain. Eight bits move accuracy down slightly, by zero point seven and zero point four points.

Four bits is where order sensitivity grows. The skew toward particular answer slots rises from zero point one five to zero point five three on the smaller model, and from zero point zero seven to zero point one two on the larger. Flip rates rise as well, but they stay below an accuracy-matched baseline of independent uniform errors.

Averaging four cyclic orders recovers twelve point six points at four bits for the smaller model, compared with five point three at fp16. To set the two spending options side by side, bits and passes are priced in one machine-independent unit: weight bytes read, times passes. In cost-matched pairs, extra orders beat extra bits only when the weights already have six or more bits.

The authors draw the boundaries themselves. Everything is simulated, on two small models, with the simplest quantizer. The six-bit gain over fp16 is reported and left unexplained. A control adds Gaussian noise at the same weight signal-to-noise ratio, and it gives slot skews of zero point one five and zero point three seven from two draws, and zero point one nine from one. That control is inconclusive on whether rounding adds slot skew beyond noise of equal power. And the claim that nothing similar came before rests on a title-level search, which found no earlier study of quantization together with option order.

Stripped down, the result is a budget rule rather than a ranking of methods. Reordering the options only beats adding bits once the weights sit at six bits or more. On these two models, that lands on six-bit weights, with whatever budget is left going to up to four orders.