Training Foundation Models with a Single Algorithm

Training Foundation Models with a Single Algorithm. By Yuda Song and nine others, from Carnegie Mellon University, Johns Hopkins University, and one other institution. Published on the twenty-ninth of October.

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning, and reinforcement learning. Moving from one stage to the next changes the data, and it changes the learning algorithm too. Each stage arrives with its own objective, so the pipeline looks like a sequence of different problems solved one after another. The argument here is that only the first of those changes is essential. The data really does differ from stage to stage. The algorithmic separation, the authors show, is not fundamental.

The key step is a derivation. Suitable reinforcement learning objectives can be written as maximum-likelihood fitting to target distributions that the model itself generates. Once that holds, every stage fits under one likelihood-based algorithm, which the authors call unified training. The training lifecycle then becomes a question of data: which data to train on, in what proportions, and when. Conventional staged training is recovered as one special case of that choice.

The framework is instantiated as a unified training system that mixes static data and reinforcement learning generations inside a single trainer, rather than handing a model from one pipeline to the next. In that setting the fixed corpus and the model's own reinforcement learning outputs are simply two sources of data feeding the same objective. Deciding how much of each to use, and at which point in training, is the remaining design choice, and the staged recipe is just one particular answer to it.

Across controlled and real-world experiments, unified training improves the tradeoff between accuracy and compute, and improves task coverage, while preserving broader capabilities. It also gives a stronger starting point for continual learning. The comparison these claims are framed against is the conventional staged pipeline, which the unified view treats as a special case of itself.

The ambition is stated plainly: groundwork for a unified science of foundation-model training, linking mathematical principles, training infrastructure and the development of capabilities. If the stages share one algorithm, the design questions that remain are all about the data schedule.