Selecting Diverse SFT Traces Improves Post-RL Generalization

Selecting Diverse SFT Traces Improves Post-RL Generalization. By Dylan Zhang and two others, from the University of Illinois Urbana-Champaign and Google. Published on the twenty-seventh of September. No code repository is linked to it.

Before a reasoning model goes through reinforcement learning, it is usually given supervised fine-tuning on verified solutions. This paper argues that those verified solutions are not equally useful as preparation. What matters is route diversity: how much the sequences of reasoning steps vary across the fine-tuning data. Choose solutions whose routes differ from one another, rather than ones that resemble each other, and the model generalizes better after reinforcement learning. To make that choice practical, the authors propose a lightweight fingerprint, built from rules, that selects for diverse routes.

The comparison is tightly controlled: one pool, one budget, matched training recipes and matched checkpoints. Under those conditions, picking diverse routes improves problem coverage after reinforcement learning, in puzzles and in mathematics, including on problems harder than anything seen in either training stage. In synthetic experiments, route-diverse fine-tuning raises the pass at eight score of OLMo3-7B by sixteen point nine points, on environments that were held out of fine-tuning. In a setting where one model writes every candidate solution, diverse selection gains as much as six point two points of mean pass at eight, across ten mathematics benchmarks.

Diagnostics taken before reinforcement learning suggest a reason. Diverse fine-tuning has slightly lower mean accuracy, but on more prompts it produces a mix of successes and failures. Group-relative reinforcement learning needs that mix to get a learning signal, so more prompts end up teaching it something. A prompt the model always solves, or never solves, gives every attempt in the group the same outcome, and leaves nothing to compare.

The selector itself runs on a CPU only and makes no model calls. On three open-source corpora it beats more expensive alternatives in every comparison of mean performance after reinforcement learning. The evidence covers puzzles and mathematics, and the headline gains are stated as maximums or as results on held-out synthetic environments, so the size of the effect elsewhere is left open.

Here is the counterintuitive part. A fine-tuning set that makes the model a little less accurate on average can leave it better prepared, because a model that sometimes succeeds and sometimes fails on a prompt is a model that reinforcement learning can still improve.