Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models

Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models. By Sai Kiran Bathula, from QLNI. Published on the fifteenth of October. No code repository is linked to it.

In a small language model, the vocabulary is the single largest part. In Gemma three, at its two hundred and seventy million parameter size, the embedding table holds one hundred and sixty-eight million of the two hundred and sixty-eight million parameters. Quantisation methods leave the embedding in sixteen-bit, so once the rest of the model is compressed, that table is also most of the file. The question here is what the table actually buys. The answer reported is less than its size suggests. One frozen table of codes, sixteen megabytes, with no trainable parameters at all, gives up a little validation loss and matches or beats the learned table on most of the downstream tasks tested.

The frozen table is derived once from a pretrained embedding, by principal-component projection and sign binarisation, which leaves a five hundred and twelve bit code for each entry. The same Gemma-shaped body is then trained from scratch on the same two point zero five billion tokens, with and without it. The learned table buys zero point two nats of validation loss, for two point six times the trainable parameters. Yet on LAMBADA, HellaSwag and PIQA, the frozen-table model is better or tied.

Swap in a ternary BitNet b1.58 body and the whole model becomes a forty-six megabyte file. It still ties the float model on LAMBADA and PIQA. On a laptop CPU it generates six hundred and fifteen to seven hundred and forty tokens per second, in sixty-two megabytes of memory. That is five to six times the speed of the float model's eight-bit GGUF in llama.cpp, at a seventh of the memory. Rare words are where the frozen table pulls clearly ahead, winning by one point seven to two point three nats. And after training, it accepts ten thousand new words at a three percent cost, where the learned table collapses by forty-three percent.

The limits sit inside the same numbers. The learned table does win on validation loss, and on ARC-Easy it is favoured outright: that is the one benchmark of the four where the frozen table falls behind. The comparison is also a single setting, one Gemma-shaped body trained on one token budget. What it shows is how the trade behaves for a small model at that scale, not how it behaves for larger models or longer training, which were not run.

Put the two halves side by side and the title's question gets a qualified yes. The learned table is the biggest thing in the model and earns a fraction of a nat for it. The frozen code table keeps the downstream scores, handles rare and newly added words better, and is what lets the ternary version fit in forty-six megabytes. The weights, the data and every evaluation answer are released with it.

Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models. By Sai Kiran Bathula, from QLNI. Published on the fifteenth of October. This is the long version: the whole paper rather than the abstract.

In a small language model, the vocabulary is the model. Gemma 3 270M has about two hundred and sixty-eight million parameters, and a hundred and sixty-eight million of them sit in one place: the embedding table, one trained vector per token, tied to the output head. Quantisation makes this worse rather than better. BitNet keeps its embedding and head in sixteen bits, and GPTQ and AWQ leave the table untouched. So once the body is compressed, the table is what is left. A ternary Gemma body would be about twenty megabytes, sitting beside three hundred and thirty-five megabytes of vocabulary. The paper's question is what that table actually buys.

A token embedding does two jobs. On the way in, it tells the body which token arrived. On the way out, it scores the body's final state against every token in the vocabulary. Neither job, the author argues, obviously needs six hundred and forty learned numbers per token. Earlier SHADOW models from the same lab shipped without a learned table, but changed many things at once. This paper is the controlled version: same body, same data, same steps, same seed, and only the vocabulary changes.

The frozen table is built once from Gemma's own pretrained embedding. Centre that matrix, find its top five hundred and twelve principal directions, project each token's vector onto them, and keep only the sign of each projection. Every token becomes a code of five hundred and twelve bits, plus or minus one, and the whole vocabulary fits in about sixteen point eight megabytes. It never changes again.

Going in, a token's code passes through a small trained linear map into the body's width. Coming out, another small map takes the body's final state back to five hundred and twelve dimensions, and that is scored against every code by a dot product, scaled by the square root of the code length. The two maps together are tiny next to the body. One more piece is added to the scores: a fixed prior, the log of each token's frequency in the training data. The paper calls it the seal. It has no trainable parameters. The codes keep a lot of what Gemma knew: the five hundred and twelve directions carry ninety-two point three percent of the variance, and on WordSim-353 the codes track human similarity judgements as well as the float embedding does. Because the table is a rule rather than a trained matrix, any vector in the embedding's space can become a new token after training.

Three arms are trained from scratch on about two billion tokens of FineWeb-Edu. Arm A is the learned tied table, Gemma as Google would train it. Arm B swaps in the frozen table and fixed seal over the same float body. Arm C keeps the frozen table but replaces every linear layer with BitNet's ternary BitLinear and lets the seal train from its starting point. A has about two hundred and sixty-eight million trainable parameters, B and C about a hundred and one million each.

The learned table buys point two nats of validation loss, about twenty-three percent lower perplexity. On the benchmarks it buys much less. The frozen table is better on LAMBADA by one point six points, tied on HellaSwag and PIQA, and worse on ARC-Easy by three point four. That is the whole return on a hundred and sixty-eight million trained parameters: one benchmark.

Splitting the loss by token frequency explains why. The learned table wins in the middle, on tokens seen between a hundred and a hundred thousand times, where there are enough examples to train a row. On the most common tokens, about half of all text, the gap is tiny. On rare tokens, seen fewer than a hundred times, the frozen table wins by over two nats, because those learned rows never got enough examples while the codes arrived already knowing the word. The loss curves agree: B starts ahead, A overtakes, and the gap then holds steady. The learned table is a capacity A grows into.

The ternary arm is the practical result. It runs as a forty-six megabyte file, ties the float learned-table model on LAMBADA and PIQA, and on a laptop CPU generates six hundred and fifteen to seven hundred and forty tokens per second in sixty-two megabytes of memory. Against the learned-table model at eight-bit, which loses nothing in perplexity, that is five to six times faster at a seventh of the memory. The learned model cannot shrink much further, because its table stays in sixteen bits.

The limitations section is direct. One seed per arm, so the error bars cover test questions rather than training runs. The data budget favours the smaller arms. Arm C differs from B in three ways, not one: the ternary body, BitNet's schedule, and the trained prior. A partial run showed the prior's share is small, but the schedule's share was not isolated. The codes come from Gemma's embedding, so the method needs some pretrained embedding to start from. The speed comparison is one laptop.

Adding words after training is reported honestly too. Ten thousand new biomedical words cost the frozen table about three percent, while the learned table, given untrained rows, collapsed by about forty-three percent. But no construction made the cost zero, so the claim is possibility, not improvement. Two tests expected to favour the frozen table, new control tokens and emoji prediction, did not.

The idea to carry away is that a vocabulary can be a rule instead of a trained matrix. Derive fixed codes once from an embedding that already exists, spend the trainable parameters on the body, and the small model loses a measurable slice of loss but very little on the benchmarks tested. The table that every quantiser leaves in sixteen bits stops being a fixed cost, and a whole language model fits in less memory than the table alone used to take.