ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning

ThinkBit. Search Codewords, Recalibrate Scales, Select by Reasoning. By Rhea Sanjay Sukthanker and one other, from Microsoft. Published on the twenty-sixth of October. No code repository is linked to it.

Reasoning language models generate thousands of tokens for every answer, and the speed of that generation is limited by memory bandwidth. Storing the weights in two bits instead of bfloat16 could therefore, potentially, make inference eight times cheaper. The catch is in how two-bit methods are usually judged: by perplexity, and by accuracy on multiple-choice question answering. Neither says much about whether the compressed model can still reason its way through harder problems.

ThinkBit is a two-bit framework built specifically to preserve reasoning. It is strictly scalar. Every weight is an integer codeword multiplied by a bfloat16 scale, and that scale is shared by a group of weights. The framework then treats the codewords and the scales as separate things to get right, in a sequence of stages, and ends by choosing between candidate checkpoints on the basis of reasoning itself.

The first stage allocates bits. A sensitivity metric assigns each layer to one of three tiers, ternary, two-bit, or three-bit, so that bitwidth is spread across layers in the best way. Next, a layerwise scalar quantization algorithm based on Gumbel-softmax searches for the best codewords, block by block.

A layerwise scheme cannot see how quantized layers interact with one another. So the third stage freezes the codewords and recalibrates the scales, using a loss based on knowledge distillation against the teacher model.

After that comes a short stage of quantization-aware training, or QAT, in which the model learns to mimic the base model's distribution by reassigning codewords. The last step is selection. A paired reasoning test on training problems picks the final checkpoint, choosing between the one from the recalibration stage and the one from the QAT stage.

On Qwen three at one point seven billion parameters, compared at matched size, the recalibrated checkpoint scores forty-six point seven on MATH five hundred. EfficientQAT, trained on the same data, scores thirty point zero. After QAT on reasoning traces, ThinkBit reaches seventy-three point zero.

At zero point five two gigabytes, that model is smaller than every published two-bit GGUF of the same model, and yet more accurate on MATH five hundred.

The framework also scales, up to fourteen billion parameters. There, a model of three point seven four gigabytes scores ninety-four point eight on MATH five hundred. The full bfloat16 model, at twenty-nine point five gigabytes, scores ninety-seven point five.

The method's argument sits in its last stage as much as its first. If the point of two-bit weights is cheap reasoning, then reasoning should be the test that decides which checkpoint ships, not perplexity or multiple-choice accuracy. Everything before that is a way of producing good candidates: bits placed where layers are sensitive, codewords searched, scales repaired once the layers are seen together, and a short training pass to close the remaining distance to the base model.