Language Models Learn to Improve Language Models

Language Models Learn to Improve Language Models. By Minhua Lin and nine others, from the Pennsylvania State University, UC Santa Cruz, and three other institutions. Published on the sixteenth of October. No code repository is linked to it.

Large language models are increasingly used to automate AI research, including the development of language models themselves, which offers one possible path toward recursive self-improvement. Turning experimental feedback into a learned ability to propose good improvements, across different models, runs into two challenges.

The first is cost. Training such an ability with real feedback through reinforcement learning means evaluating tens of thousands of proposals, and each one may need a full model-training run. Even at ten GPU-hours per run, collecting the rewards alone would take hundreds of thousands of GPU-hours. A cheaper surrogate is needed. The second challenge follows from the first. A surrogate can become unreliable, because policy optimization tends to favour proposals the surrogate overestimates, especially once the distribution of proposals shifts away from what it has seen.

LM2LM uses reinforcement learning to train language models to generate effective proposals for improving language models. Its reward comes from a research world model, or RWM. The RWM is a frozen language model that predicts how much a proposal will gain, by reading real experimental records supplied in its context. It stands in for running every experiment.

To keep that surrogate honest as training goes on, LM2LM periodically takes proposals from the current policy that the RWM ranks highly, actually executes them, and adds the measured outcomes to the RWM's context. The predictor is thereby confronted with real results in exactly the region where the policy is now searching, which is where overestimation would otherwise creep in. Real compute goes only to the proposals the RWM ranks highest, and the rest of the feedback stays cheap.

The evaluation spans eight development environments covering both autoregressive and diffusion language models, with real-execution evaluations on held-out tasks. There, LM2LM enables Qwen3.8-27B to outperform GPT-6-sol. That holds even when GPT-6-sol is given three hundred real experimental records in context, and LM2LM does it at thirty-three percent lower data-generation cost. It also generalizes to a model family not seen during training.

The method rests on a division of labour between imagination and measurement. A frozen model predicts outcomes cheaply from records, the policy learns against those predictions, and a steady trickle of real experiments keeps the predictor anchored where the policy is heading. Without the trickle, the surrogate would be gamed; without the surrogate, the training would be unaffordable.