Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model

Precision on Demand. Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model. By Mohammed Nayeem Khan and two others, from Intel Corporation. Published on the tenth of November.

Local decoding usually fixes one weight precision and keeps it for the whole generation. Every step then pays the same price, whether the model is sure of its next token or not. The study looks at a different allocation, where precision is spent unevenly, one step at a time.

Store the weights at eight bits, but decode with only their top four bits. At each step, inspect the output distribution of that low-precision pass. If it looks uncertain, redo the step at the full eight bits. Uncertainty is judged in one of two ways: entropy above a threshold, or a small margin between the top two candidates.

Quantization is simulated, the model is Qwen3-0.6B, and the benchmark is GSM8K. Cost is counted in weight bytes read per token. The first test is run on thirty-six thousand seven hundred and twenty-four replayed steps, and asks whether the gates find the steps where low precision changes the top choice, the so-called argmax flips.

They do. The entropy of the low pass ranks those flips with an AUC of zero point nine three. At a fifteen percent escalation budget, the entropy gate catches seventy-eight percent of flips, and the margin gate ninety-four percent. A random gate escalating at the same rate would be expected to catch fifteen percent, a figure computed rather than run.

Then come one hundred and fifty live problems. Running everything at high precision scores sixty-four point zero percent, and everything at low precision fifty-five point three, with a McNemar p-value of zero point zero five three. An entropy gate escalating sixteen percent of the time scores sixty-two point seven percent, at zero point eight seven of the all-high weight bytes.

Against all-high, the gate's paired difference is minus one point three points, in a range from minus eight to plus five point three. Against all-low it is plus seven point three, in a range from zero to fourteen point seven, with a p-value of zero point zero eight zero. The live data therefore do not separate the gate from either end. In the same cost model, verified speculation reaches zero KL at similar bytes.

Offline, uncertainty in the cheap pass does point at the steps where cheapness changes the answer. Whether that carries through to accuracy is not settled by the live run. A live random control, a thinking-mode phase rule, a second model and live KL were not run, and those are the pieces that would decide it.