Learning to Learn a Language

Learning to Learn a Language. By Lennart Carstens-Behrens and one other, from the Fraunhofer Institute for Algorithms and Scientific Computing and University Hospital Bonn. Published on the fifth of October. Its code repository is public and real: about twenty source files.

A language model normally learns a language by reading a great deal of it. This work asks for something stranger: a model that has never seen a word of any real language, and yet, given a prefix of real text, predicts that language in context with its weights frozen. Whatever it knows about the language in front of it would have to come from the prefix alone.

The Prior-Fitted Language Model, or PFLM, is a three hundred million parameter byte-level transformer, pretrained only on samples from a synthetic, non-linguistic prior. Every training sequence comes from a recurrent structural causal model, drawn fresh from a distribution over such models. The model therefore never sees the same language twice. The only way to predict a continuation is to infer the language from the prefix.

The prior is not language, but it is not arbitrary noise either. Samples from it share the statistical signatures of natural text: Zipfian frequencies, a slow convergence of the entropy rate, and long-range dependence. Those are properties real text has, supplied here without any real words, so a model trained on them meets familiar statistics when it is finally shown text.

On Wikipedia in six languages, bits per byte fall from the uniform eight to between zero point nine and two point four, at one million bytes of context. Given numerals instead of text, PFLM learns to count, to compare magnitudes, and to add approximately. It predicts deterministic sequences such as Rudin-Shapiro and the prime indicator. And it compresses six non-text domains, ranging from source code to speech, below gzip and PPMd.

The abstract ends on a distinction that is the whole paper in two sentences. The model has not learned a language. It has learned to learn one. Every language it handles, real or synthetic, numerical or spoken, it meets for the first time in its context window, and the pretraining is what taught it how to make sense of one there.