N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models

N-Gram Memories at Two Bits. Quantizing Lookup Tables in Large Language Models. By Donghoon Han and four others, from Dnotitia and KAIST. Published on the sixteenth of October. No code repository is linked to it.

Conditional memory gives a language model more capacity through selective lookup: an input fetches only the rows of an embedding table that it needs. The catch is that the full table must stay available, because any possible input might ask for any row. The memory is read sparsely but has to be kept whole.

Conditional Program, or CP, is a compressed representation built so that only the rows an input requests are reconstructed. Each row is stored as a short code plus a local scale, and a single reconstruction program is shared across the entire table.

The authors instantiate this with three components: a Gaussian trellis, norm restoration, and row records that can each be decoded independently. Analysis across three trained families of n-gram memory supports the design. Their coordinate statistics look Gaussian, yet learned relationships among rows coexist with that. At matched storage budgets, the shared trellis outperforms both the fitted codebooks and the low-rank residual variants the authors tested.

At two bits, CP's code lowers output KL by between forty-one and sixty-three percent, relative to two-bit round-to-nearest. For DeepSeek, table storage falls from two hundred and three gigabytes to fifty-one gigabytes, which leaves room for larger workloads when the table is kept on the GPU.

The bet is that a lookup table does not need to be stored as a table. Keep a short recipe per row and one shared program, and the memory shrinks to roughly a quarter of its size while each row stays individually reachable on request.