Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization

Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization. By Jiacheng Liu and seven others, from the Mohamed bin Zayed University of Artificial Intelligence, Westlake University, and three other institutions. Published on the twenty-second of October. Its code repository is worked out: about eighty-five source files, with a readme and a licence.

Language models are increasingly scaling their embedding capacity alongside Transformer computation, and that trend makes high compression rates more and more useful. Per-Layer Embeddings, or PLE, follow the trend closely: they account for thirty-five point two percent of the parameters of Gemma 4 E4B. The paper studies compressing them to an extreme degree. At such rates, conventional quantizers give each token-layer vector its own code, and that code ends up only a few bytes long. Spread that thin, there is very little left for any single vector.

OPTIC, short for Output-Path-fitted, Token-shared Indexed Codes, changes what receives a code. Instead of one code per token for every layer, it stores a single quantized code tuple per token, and decodes that same tuple at every layer with a layer-specific affine decoder. The token owns the code. Each layer differs only in how it reads it.

The second change concerns what the fitting aims at. Codes and decoders are fitted to the PLE output path, rather than to the stored vectors themselves. The target, in other words, is the path the embeddings feed into, not a faithful copy of the tables as they sit in memory. Both choices work toward the same end: spending a tiny budget on what the model actually uses.

At about one hundred times PLE compression of bfloat16 E4B, OPTIC retains an aggregate ninety-seven point eight percent of the uncompressed model's performance across seven benchmarks. Product quantization, the strongest of six conventional baselines, retains sixty-six point three percent. Pushed further, to five hundred and ninety-four times compression, OPTIC retains ninety point five percent. At one thousand and two times, it retains eighty-four point one percent.

The method also holds on the official E4B release, which uses four-bit linear weights and sixteen-bit activations. There, OPTIC retains ninety-six point nine percent at about one hundred times compression. During inference at that same rate, bfloat16 E4B uses twenty-nine point seven percent less GPU memory than it does with uncompressed PLE at the same key-value cache size, while running at most zero point four percent slower.

Two decisions carry the result. One code per token rather than per token and layer, so the budget is spent once and reused everywhere. And a fitting target set by the output path rather than by the stored vectors. Together they let a block holding over a third of the model's parameters shrink by two to three orders of magnitude, while most of its benchmark performance survives.

Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization. By Jiacheng Liu and seven others, from the Mohamed bin Zayed University of Artificial Intelligence, Westlake University, and three other institutions. Published on the twenty-second of October. This is the long version: the whole paper rather than the abstract.

Language models are starting to scale their embedding tables alongside their computation, and Gemma 4 is one example. Its Per-Layer Embeddings give every vocabulary item a learned vector at every decoder layer, not just at the input. In the E4B model those tables are thirty-five point two percent of all parameters. The question here is how far that can be squeezed, and the difficulty is arithmetic. E4B stores eleven million token and layer vectors, each two hundred and fifty-six numbers wide. At about one hundred times compression, that leaves roughly five bytes per vector. Conventional quantizers spend those five bytes on a separate tiny code for every vector, and at that size a tiny code is not enough.

Two facts about these embeddings set up the method. The first is their shape. The table is indexed by token and by layer, so a compact form could share storage along either axis.

The second is how the model uses them. A layer never adds its embedding vector straight into the hidden state. It multiplies the vector element by element by a gate computed from the hidden state, then projects the product up to the hidden width through a frozen output matrix. Error in a coordinate that the gate suppresses, or the projection ignores, costs nothing. Error in a coordinate the projection amplifies costs a lot. Measuring reconstruction error in the stored coordinates treats all of them alike.

The method is called OPTIC, short for Output-Path-fitted, Token-shared Indexed Codes. Each token stores one tuple of codebook indices. In the configuration near one hundred times, that is one hundred and sixty-one one-byte indices, each picking a four-number codeword from its own codebook of two hundred and fifty-six entries. Concatenated, they form one latent vector of width six hundred and forty-four. Each layer then has its own affine decoder: a matrix and a bias that turn that shared latent into that layer's embedding row. Storage becomes token codes, plus codebooks, plus forty-two decoders. Sharing codes removes the layer factor from the biggest term, and the decoders do not grow with the vocabulary. For E4B the whole thing comes to about fifty-six point four megabytes, against five point six four gigabytes uncompressed. Decoding a token gathers its codewords and applies one decoder, without ever rebuilding the full table.

The second idea is what to fit against. Using cached gates from the uncompressed model on natural text, plus one synthetic record with all-ones gates for every token in the vocabulary, the loss measures the change a reconstruction causes at each layer's projection output, before normalization and the residual add. Fitting alternates. With codes fixed, gradient steps update the codebooks and decoders against that output-path loss. Then every token's codes are reassigned, one codebook group at a time, under a surrogate that averages the gate statistics into one weighting matrix per layer. That averaging matters because many tokens never appear in the fitting text, yet all of them need codes. Initialization uses the same weighting: a weighted principal component analysis of each token's concatenated layer rows, then K-means within each group.

Retention is the geometric mean, over seven benchmarks, of each score relative to the uncompressed model, capped at one, so a single zero task zeroes the whole. Near one hundred times on BF16 E4B, OPTIC retains ninety-seven point eight percent. Product quantization, the best of six conventional baselines, retains sixty-six point three. On GSM8K, OPTIC scores ninety-two point eight against ninety-two point nine uncompressed; product quantization falls to seventy-three point five. At five hundred and ninety-four times and one thousand and two times, OPTIC still retains ninety point five and eighty-four point one, while the authors' own output-aware method without code sharing collapses to zero.

The comparison that most directly tests the thesis is quieter. Independent latent codes keep everything from OPTIC's recipe, including the output-path fitting and nearly equal bytes, but give every token and layer pair its own code. They retain seventy-nine point three, against ninety-seven point eight with sharing.

Two more findings appear only in the body. The table really is token-specific: replacing every vector with its layer's mean drops GSM8K from ninety-two point nine to forty-three point five. And under the output-path weighting, the variation is far less uniform. Capturing ninety percent of a layer's variation takes two hundred and twelve to two hundred and twenty-seven of two hundred and fifty-six directions in stored coordinates, but anywhere from one to two hundred and five, median one hundred and fifty-eight, at the projection output. Serving on one RTX 4090 with a fixed cache, BF16 E4B uses twenty-nine point seven percent less GPU memory and runs at most zero point four percent slower.

There is a limitations section, and it is specific. Only Gemma 4 E2B and E4B are studied. The main results use one fitting seed, so the bootstrap intervals cover benchmark sampling, not refitting. The output-path loss holds gates fixed and looks at one layer's projection at a time. The baselines are the authors' adaptations to this setting. And one low-count task can swing retention strongly: AIME has only thirty problems. Gains from fitting choices are also modest beside the representation: the full output-path objective beats Euclidean fitting by about two points, and diagonal weighting captures most of that.

The idea to carry away is that a table indexed two ways invites sharing along both. OPTIC gives each token one rich code and each layer one cheap decoder, and fits them where the model actually feels the error. The result is a table a hundred times smaller that the model barely notices.