The Lexical Imprint of Pre-training Data

The Lexical Imprint of Pre-training Data. By Jeonghoon Kim and five others, from NAVER Cloud and KAIST. Published on the twenty-sixth of October.

Here is the question this work sets out to answer. When a base model scores well on a zero-shot benchmark, how much of that accuracy is simply captured by word-level alignment between the corpus it was pre-trained on and the benchmark's target text? The answer the authors give is that a great deal of it tracks this alignment, and consistently. Their reasoning starts from how such models are trained and scored. Likelihood training shapes a model's predictive distribution from its pre-training corpus. Zero-shot scores under likelihood-based evaluation then depend, in part, on how lexically close that corpus is to the benchmark target. The corpus, in other words, leaves an imprint.

To measure that closeness they use word-level unigram cross-entropy, shortened to UCE. It is a tokenizer-agnostic and model-free measure of lexical alignment between a reference corpus and a target benchmark. Lower UCE means the corpus is more aligned with the benchmark. Because it needs no model and no tokenizer, it can be computed before any training happens.

Tracing the imprint cleanly needs a controlled setting, one that varies which corpus a model is pre-trained on while holding the training recipe fixed. So the authors pre-train from scratch across a sweep. It spans ten zero-shot benchmarks, four pre-training corpora and three model scales. Across that sweep, UCE consistently tracks zero-shot accuracy on every benchmark. Corpora more aligned with a benchmark, the ones with lower UCE, reach higher accuracy.

The signal also moves from measurement to control, at finer scopes. At the document level, the authors strengthen the imprint by selecting training documents that maximize lexical alignment with the benchmark, and this produces the accuracy gains the mechanism predicts. At the prompt level, they match the imprint per prompt instead. Each prompt is routed to the drafter whose domain corpus is most lexically aligned with it, and this yields measurable throughput gains over a baseline that uses a single generalist drafter.

The claim has clear edges. It concerns base models and their zero-shot behaviour, measured under likelihood-based evaluation, which is the setting where the authors' reasoning about predictive distributions applies. The central evidence comes from one controlled sweep of four corpora, ten benchmarks and three scales, trained from scratch with a fixed recipe. And the authors' framing is that zero-shot scores depend on lexical alignment in part. Nothing here says alignment is the whole of what a benchmark measures.

Two things come out of this. One is a picture: pre-training corpora leave a measurable lexical imprint on how base models behave zero-shot. The other is a tool. UCE gives a cheap, model-free probe of how lexically aligned a reference corpus is with target data, usable for diagnosing benchmark scores and, as the document and prompt experiments show, for steering them.