Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale

Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale. By A. Bochkov. Published on the second of October. No code repository is linked to it.

A trainable input embedding table gives every item in the vocabulary its own vector, adjustable independently of all the others. The question here is whether substantial language-modeling ability actually requires that token-specific parameterization, or whether a shared Transformer can learn from token identities that are fixed and never trained. The answer offered is that the trainable vectors are not required for the capabilities observed. Models whose input is a fixed code still learn a great deal.

Three decoder-only models are trained from scratch, identical in everything but their input. They share the tokenizer, the contextual backbone, an untied output-head architecture and the training recipe, with a target budget of one hundred billion prediction tokens each. The first has a learned table. The second uses canonical sixteen-bit token-ID codes. The third uses one fixed, invertible recoding over GF two. The fixed codes are simply repeated out to the model's width, with no trainable input projection added.

Both fixed-code models acquire substantial capabilities. With canonical codes, HellaSwag normalized accuracy is fifty-two point four percent. PIQA accuracy is seventy point five one percent. LAMBADA accuracy is forty-two point seven five percent. Dropping the trainable table removes one hundred point seven million trainable parameters, leaving models of one point seven one one billion parameters.

The paper is careful about what this does not show. The learned-input control does better on several evaluations, HellaSwag and LAMBADA among them, so the result is viability, not parity. The parameter saving is real but is not presented as the central point. These are single-run experiments. What they separate is architectural necessity from empirical utility: a trainable table may still help, but it is not needed for these capabilities to appear. And while a fixed identity interface gives a controlled setting for studying representation learning downstream of an input that cannot change, the paper does not establish where particular capabilities are localized.

The finding is narrow and deliberately so. It moves the input embedding table from the column of things a language model must have into the column of things that may help. How much it helps at this scale, and where inside the model the observed capabilities actually live, are left as open questions.