Learning Latent Protein Languages for Autoregressive Generation

Learning Latent Protein Languages for Autoregressive Generation. By Mahdi Pourmirzaei and four others, from the University of Missouri, the University of Southern California, and one other institution. Published on the second of October. Its code repository is public and real: about thirty source files.

Autoregressive transformers have stayed comparatively weak at generating protein sequences and protein structures. The authors look for part of the reason in the target representation, meaning what the model is asked to predict. Amino acid tokens encode which residue is where, but carry no explicit contextual semantics. Backbone coordinates have the opposite difficulty: in this framework they need a discrete representation before an autoregressive model can predict them at all. One target is too thin in meaning, the other is not a sequence of tokens to begin with. Either way, the question becomes what alphabet a protein should be written in before a transformer is asked to continue it.

Their answer is to learn the vocabulary. Two latent protein languages are introduced. Protein Latent Language, PLL, maps a sequence onto a contextual alphabet of four thousand and ninety-six states, built on a frozen ESM-2 encoder, with one token per residue. Structure Latent Language, SLL, adapts GCP-VQVAE Lite, adding auxiliary sequence and confidence supervision while keeping the ability to decode back to backbone coordinates. The first alphabet is about meaning: each residue token carries context from the encoder rather than identity alone. The second is about geometry: a discrete code for structure that can still be turned back into coordinates.

With the two alphabets in hand, autoregressive transformers are pretrained separately on each, using plain next-token prediction. The model trained on PLL tokens is called PLLM. The one trained on SLL tokens is called SLLM. Nothing about the transformer recipe is new. The change is entirely in the tokens.

Start with sequences. Under matched downstream sequence training, PLLM has a fitted compute-scaling exponent of zero point zero three eight, against zero point zero two zero for the amino acid autoregressive model. In unconditional sequence generation, there is a heuristic threshold of one point five bits of residue-composition entropy, and PLLM reduces the fraction of samples falling below it by fifty-four percent relative to the amino acid model, across sampling temperatures. A higher scaling exponent means more improvement per unit of compute, and fewer samples under the entropy threshold means fewer generations with low residue-composition entropy.

Then structure. For sequence-to-structure prediction, replacing the original GCP-VQVAE Lite tokenizer with SLL reduces best validation perplexity by thirty-four percent under matched training. For long proteins, latent-token sampling is roughly one thousand times faster than MSA-based AlphaFold2, in the authors' measurements. In backbone generation, SLLM compares favorably with other generative models on diversity and novelty.

One more result is labelled as early. Using SLLM's internal token confidence for sampling at inference time shows signs of lifting sequence-to-structure prediction quality beyond what a single decoded sample gives. In other words, the model's confidence in its own tokens may be usable to pick among samples, though the authors present this only as a first sign.

The authors position learned latent protein languages as a promising substrate for autoregressive transformer scaling and for inference-time sampling in protein generation. The thread through every result is that the architecture held still. What moved was the alphabet the model was asked to write in.