Follow the Gradient: Tracing Safety Degradation Back to Training Data

Follow the Gradient. Tracing Safety Degradation Back to Training Data. By Saket Reddy and five others, from Datology AI and the University of Illinois Urbana-Champaign. Published on the twenty-seventh of October.

Fine-tuning a large language model on instruction data can quietly wear away its safety, even when not a single training example looks harmful. That is the whole difficulty. If the data doing the damage reads as benign, a standard content filter struggles to catch it, because the filter is judging what each example says, and what each example says is unremarkable. Whatever degrades safety has to be found some other way, and it has to be found across a full post-training dataset rather than a hand-picked sample of one.

SimpleLoRIF looks at gradients instead of text. It is a scalable gradient embedding method, and what it identifies are fine-tuning examples whose gradients align with unsafe behaviour. An example may say nothing alarming at all, but if learning from it moves the model in the same direction that unsafe responses do, that alignment is visible in gradient space. Gradient attribution has mostly been a small-scale analysis tool. The aim here is to turn it into a practical filter for whole post-training datasets.

Two simplifications buy the scale. SimpleLoRIF simplifies an existing low-rank attribution method, and it restricts backpropagation to the final four transformer blocks of the model. With both in place it can embed datasets on the order of a million examples, which is the size that a filter for real post-training data has to handle.

Validation starts with emergent misalignment. There, gradients connect insecure-code training data to the broadly harmful behaviours that appear after fine-tuning, with an AUC as high as zero point eight six five in the gradient space of misaligned models. The method is then put to work. SafeTulu is a version of Tulu 3 with the twenty percent of examples most gradient-aligned with unsafe responses removed.

Training on SafeTulu reduces the success rate of the GCG attack by up to twenty-two percent across Qwen, Llama, and Olmo. It outperforms both an LLM-as-a-judge filter and prior gradient-based filtering, while largely preserving capability and in places improving it. An analysis of the removed data then points to possibly under-studied sources of safety degradation, such as tool-execution failures and narrative arithmetic.

SafeTulu is released along with a Hugging Face Space where researchers can build their own customised versions to fine-tune on, changing the removal fraction or filtering only certain subsets. Underneath the release is a change in where to look. The examples that erode safety do not look harmful, so their text is the wrong place to search. Their gradients are where the signal shows up, and once those can be computed for a million examples, attribution stops being a diagnosis after the fact and becomes a step before training.

Follow the Gradient. Tracing Safety Degradation Back to Training Data. By Saket Reddy and five others, from Datology AI and the University of Illinois Urbana-Champaign. Published on the twenty-seventh of October. This is the long version: the whole paper rather than the abstract.

Fine-tuning a language model on ordinary instruction data can quietly wear away its safety. Earlier work showed that models trained on widely used, benign datasets become noticeably easier to jailbreak, and that the damage comes from a small subset of the data rather than from all of it. That turns safety into a data selection problem: find the responsible examples and remove them before training. The trouble is that those examples look harmless: lists, bullet points and math questions. Toxicity filters and language-model safety judges read what a document says, and what it says is fine. The proposal is to stop reading the content and ask instead what the document would do to the model.

What a document would do to a model is its gradient: the update the model would make by training on it. Two documents that share no words can push the weights in the same direction. Gradient attribution, in the line of TracIn, scores a training example by how closely its gradient aligns with the gradient of some target behaviour, here unsafe behaviour. The obstacle has always been cost. A full per-example gradient is as large as the model, and curvature-corrected influence functions are intractable at this scale. So gradient methods have stayed analysis tools for small datasets.

The method, SimpleLoRIF, builds on an existing scalable influence method called LoRIF, in three moves. The first is compression. A linear layer's weight gradient is already a sum, over tokens, of input activations multiplied by output gradients. So fixed random projections can be applied to those two vectors as they emerge from backpropagation, and the full gradient is never formed. Each layer's gradient shrinks to a thirty-two by thirty-two matrix. Because per-example gradients have low effective rank, only its leading rank-one factor is kept: two short vectors per layer, sixty-four numbers instead of one thousand and twenty-four. Comparing two documents multiplies how well their first vectors agree by how well their second vectors agree, summed over layers, which equals the alignment of the two compressed gradients without rebuilding them.

The second move is a subtraction. LoRIF adds a curvature correction so its scores estimate real loss changes. Here the score only ranks documents for removal, so the correction is dropped.

The third move is depth. With curvature gone, the dominant cost is the backward pass. Of four knobs the authors ablated, only one paid off: restricting backpropagation to the top few transformer blocks kept quality, because the backward pass can stop early. Two blocks dropped to point six five AUC, six or eight gained little, and four was chosen, for a ninety-one point five percent speedup. With that, all nine hundred and thirty-nine thousand Tulu 3 examples were embedded in about an hour on eight H100 GPUs, using a small Qwen 3 model.

The target set is the unsafe responses in SafeRLHF. Rather than one pooled target, it is clustered, and each training document is scored against the cluster centres, which the rank-one factors allow exactly. The top twenty percent are removed. What remains is SafeTulu.

The first test is emergent misalignment, where fine-tuning on insecure code makes a model broadly harmful on unrelated questions. Insecure and matched secure code examples were scored against harmful answers the misaligned model itself gave. Insecure code ranked higher, with an AUC of point six oh four for Qwen and point eight six five for Mistral. The appendix adds the control: before the insecure fine-tuning, the same analysis gives point five five two and point four three one. The connection appears only once the model has gone wrong.

On Tulu, three small models were each trained with three seeds. Under the white-box GCG attack, Qwen's attack success fell from forty-seven point seven percent to thirty-seven, where the language-model judge filter managed forty-seven point three. Llama and Olmo moved the same way, and SafeTulu also had the lowest success rate for the multi-turn ActorBreaker attack on every model. Capability mostly stayed close to full-data training.

The comparison that tests the thesis most directly is a sweep over how much to remove, run on Llama. The judge filter helps up to ten percent removal, then gets worse than not filtering at all: thirty-nine percent success at twenty, forty-three at forty. Gradient filtering keeps improving through twenty percent, down to thirty point three, before it too turns. The reading offered is that a content judge runs out of genuinely degrading data quickly and starts discarding data that may support safety. Gradients find a pool roughly twice as large. The removal pattern agrees: the judge cut three math subsets at almost identical rates, while gradient filtering ranged from seven point nine to nineteen point four percent across them.

There is a short limitations section, and it names two things. Only general safety degradation was tested, not specific harms or behaviours such as sycophancy. And only one target set, SafeRLHF, was tried. The evidence has further edges. The fine-tuned models are small, and the removal sweep used one model. The headline reduction of up to twenty-two percent is relative, and is the best case. And the qualitative findings, that Python tracebacks, simple coding tasks and arithmetic carried through prose are removed more often, come from a correlational word analysis. The authors stress these patterns are suggestive only, and make no claim that they cause safety degradation.

The idea to carry away is that the unit of data safety is the update, not the text. A document can be spotless and still push a model toward compliance, and the only place that shows is in the direction it moves the weights. What this work contributes is making that question affordable at the scale of a full post-training mixture. The open question it leaves is causal: remove only the tracebacks, retrain, and see whether attack success actually drops.