Selection Is Not Learning: Choosing Model Futures

Selection Is Not Learning. Choosing Model Futures. By Haoyi Yang and three others, from the Mohamed bin Zayed University of Artificial Intelligence. Published on the fifteenth of October. No code repository is linked to it.

Self-evolving language models train on tasks they generate themselves, so every round has to decide which updated model the next round inherits. Current loops, among them R-Zero, Agent0 and Skill Self-Play, make that decision through the task. They admit tasks that look learnable, and they inherit every update those tasks produce. The authors identify a selection-to-update gap in this design. A task that looks learnable need not yield a useful update. Indeed, most updates on verified tasks fail to make correct answers more likely, and yet the loops keep every one of them. The check happens at the wrong point. It asks whether a task looks promising before training on it, and never asks whether the training that followed actually helped.

The title puts the fix in four words: selection is not learning. Choosing a good task is one decision. Choosing which model to carry forward is another, and it should be judged on the update itself, after it has happened, rather than inferred from how promising the task looked.

Two mechanisms carry this out. HeirCommit freezes the updated model and uses fresh evaluation questions to decide between inheriting it and keeping the current model. EvoHeir goes further, separating two kinds of inheritance: parameters and experience. It draws on current experience and on archived experience to repair failures and preserve successes, before a final confirmation. On the analytical side, the paper characterizes selection-induced optimism, and decomposes regret into three kinds of error: selection, absorption and confirmation. Put plainly, the two mechanisms answer two different questions. HeirCommit asks whether a new model has earned its place. EvoHeir asks what deserves to be kept from a round, whichever model ends up carrying it forward.

HeirCommit cuts the harmful share of accepted updates by sixty-eight to eighty-three percent, measured against random inheritance at the same acceptance rate. Across four backbones, EvoHeir achieves the best averages in both mathematics and general reasoning. On the math average it gains five point three percent relative to the strongest baseline, which is Agent0, one of the very loops whose inheritance rule the paper questions.

Separating parameters from experience changes what a loop is able to keep. Under this framework, useful experience persists even when the model that produced it is not inherited. A rejected update no longer has to take everything it encountered down with it.