Identically Trained Language Models Break on Different Jailbreaks

Identically Trained Language Models Break on Different Jailbreaks. By Chi-Wei Chang and three others, from Cerulean Labs, National Taiwan University, and one other institution. Published on the twenty-fifth of October.

Jailbreak benchmarks test a single checkpoint, and they treat the set of jailbreaks that break it as a property of that model's pretraining data and its safety training. This work puts a direct question to that assumption. Run pretraining again on the same data, apply the same safety training, and ask whether the new model breaks on the same jailbreaks. Mostly, it does not. Which jailbreaks break a checkpoint depends much more on its particular pretraining run than on the data that run was given.

The experiment is built to pull those sources apart. For each of twenty-five pretraining data recipes, the authors take one billion parameter base models from three pretraining seeds. Each receives the same safety post-training under two post-training seeds, which makes a hundred and fifty models in all. Every model gives twenty responses to a shared set of jailbreak prompts, and the per-prompt non-refusal rates are correlated between every pair of models.

On average, two post-trained copies of the same pretraining run correlate at zero point nine eight four. Swap in a different pretraining seed and the correlation falls to zero point eight one one. Change the whole data recipe as well, and it drops only slightly further, to zero point seven seven nine. Nearly all of the divergence arrives with the seed. Put the other way round, two models trained on entirely different data recipes agree almost as well as two models trained on the same recipe with different seeds, and far less well than two safety-tuned copies of one pretraining run. The same shows up in attack selection: of ninety-one attack templates, the best one found on a checkpoint stays the best after a new pretraining seed in only twenty-seven percent of model pairs, against sixty-seven percent after a new post-training seed.

How far does this reach? The pattern holds in every one of the twenty-five data recipes, and it holds at smaller scales too, at a hundred and fifty million and at three hundred million parameters. Those are the scales the abstract reports, with the one billion parameter models as the largest. The claim is also specific in what it is about: which prompts succeed against a model, measured as per-prompt non-refusal rates, rather than how often a model refuses overall. Nor does it say the data and the safety training are irrelevant, only that the pretraining run itself carries most of the weight.

Perhaps the most striking consequence is that a pretraining run leaves a fingerprint. Across three thousand two hundred and eleven prompts, a model's per-prompt non-refusal rates identify its pretraining run among all seventy-five runs every time. A random subset of just fifty prompts still does so ninety-eight percent of the time. A jailbreak result on one checkpoint, then, says a good deal about that run, and much less about the recipe it shares with its siblings.