How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text. By Jenna Russell and six others, from the University of Maryland and Pangram Labs. Published on the thirtieth of September. Its code repository is worked out: about one hundred and eighty source files, with a readme and a licence.

Web text is most of what language models are pretrained on, and more and more of it is written by AI. After FineWeb quality filtering, the authors find that twenty-seven point five percent of tokens from June web data are labelled as AI-generated by Pangram, rising to thirty-one point one percent by August. This is not synthetic data or a model-collapse experiment. This wild AI text comes from many models, is written for human readers, and arrives in pretraining corpora unlabelled.

The finding is that its value is not fixed, and can change sign. For a model starved of data, adding AI tokens first lowers loss on human text, but that benefit saturates and quickly reverses into harm. For a model already trained on a large budget of human text, AI tokens raise loss almost immediately, while the same number of fresh human tokens would keep lowering it.

To get there, the authors pretrain eight hundred language models, varying the ratio of added AI tokens to human tokens, and fit scaling laws to held-out loss on both human text and AI-generated text. Existing scaling laws, such as the one by Hoffman and colleagues, fail to predict the behaviour they observe.

So they propose a new law with two separate terms, one for benefit and one for harm. That structure lets the value of an AI token change sign, and the law reduces to Chinchilla when no AI text is present. Fitted on smaller models, it predicts the effect of AI text on held-out human-text loss for models up to three point six times larger, with forty-one percent lower error than the best existing law, across all AI ratios.

The effect depends on the target. Everything above concerns loss on human text, and the authors are explicit that AI text remains valuable when the target is AI text. It also depends on regime: data-starved models see an initial benefit, while models with a high human-text budget see harm almost at once. The prediction claim is bounded too, extrapolating from smaller models to ones up to three point six times larger.

From this the authors draw three recommendations. Filter AI text when the target is human text. Repeat human text before expanding a training set with AI-generated web text. And report validation loss on human and AI text separately. They release WildAI, an eighty-three billion token corpus with AI, topic and format labels, along with all eight hundred models and the code.

The title asks how much an AI token is worth, and the answer here is that it depends on how much human text came before it. Early on it can help. Past a point it costs, while a fresh human token would still have paid. With nearly a third of filtered web tokens already labelled as AI, the question of which side of that point a model is on is no longer hypothetical.