Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior. By Atsuki Yamaguchi and five others, from the University of Sheffield, the Mohamed bin Zayed University of Artificial Intelligence, and four other institutions. Published on the thirtieth of September. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.

Before a language model is pre-trained on real text, it can first be trained on synthetic data that is not natural language at all. This pre-pretraining is known to make the later pre-training more token-efficient. Prior work explained the gain as a grammatical prior: a structural inductive bias, learned in the synthetic phase, that transfers to the grammar of natural language. The title states the finding in two halves. The benefit survives at scale. The grammatical explanation does not. Those two halves pull apart a result and its reason. On this evidence the practice holds up, while the story told about why it works needs replacing, and the study offers a different reason in its place.

Until now, pre-pretraining had been tested only on models of at most one billion parameters, with pre-training budgets under two billion tokens, mostly of web text. Whether it would hold up larger, and under realistic mixtures that combine sources such as code and math, was unknown. The study here spans five synthetic tasks, four pre-training data mixtures, and four model scales from five hundred million to seven billion parameters, with pre-training budgets reaching one hundred billion tokens. That widens the earlier tests on every axis at once: larger models, far more tokens, and data mixtures closer to what is used in practice.

The gains in downstream performance and token efficiency persist at those scales. At three billion parameters, for instance, pre-pretraining saves at least twenty-one billion pre-training tokens. The grammatical story, though, finds no consistent support. Downstream performance does not line up consistently with grammatical acceptability across model sizes. What does line up is long-range retrieval: the gains come from the synthetic tasks that improve it.

The effect is also robust to how the pre-training mixture is put together. It holds across the mixtures tested, and fades only when web text is left out entirely. The negative result is specific, too. The authors do not find that grammar plays no role anywhere. They find no consistent evidence that the gains come from it, which is the explanation prior work had offered. Web text, notably, is the one ingredient whose absence makes the benefit shrink, so the result is tied to mixtures that include it.

For a practitioner the upshot is two-sided. Pre-pretraining is a low-cost addition to pre-training that keeps paying off as models grow. And anyone designing the synthetic tasks should aim them at long-range retrieval, rather than at the grammar of natural language, since that is where the measured benefit actually comes from.