Representation Redistribution in Sparse Autoencoders

Representation Redistribution in Sparse Autoencoders. By Ziye Mei and one other, from the University of Wisconsin–Madison. Published on the fifteenth of October. Its code repository is public and real: ten source files.

Sparse autoencoders decompose a language model's activations into sparse features. But a prompt can be transformed in ways meant to preserve the underlying request, and those transformations can still redistribute activity across the features. That raises two difficulties for anyone reading the features. A change seen on one transformed prompt need not recur on the next. And even a change that does recur need not affect any downstream computation. Treating every feature change as meaningful, or every recurring change as a failure, would overstate what is going on.

The authors introduce the redistribution sensitivity score, RSS. It picks out large feature responses that recur in a consistent direction, and the supports it selects are then evaluated on variants that were excluded from selection, so that recurrence is tested rather than assumed.

The second step separates three things that are easy to run together: recurring representation change, the signed contribution that change makes to a readout, and an actual decision failure. For a fixed linear readout, an exact decomposition shows how four quantities determine the downstream consequence. Those are the magnitude of the response, the readout coefficients, directional alignment, and the clean margin. Representation change is only one ingredient of a failure, not the failure itself.

The scale is twenty-seven pretrained sparse autoencoders, spanning eight language models. Across them, the tested prompt transformations produce a reproducible redistribution profile on the excluded variants.

The downstream test uses two frozen safety readouts. Combining RSS support erosion with the full clean margin yields recurrent failure AUROCs of zero point nine three five and zero point nine five eight. Those use median supports of only twenty and nineteen features respectively. For comparison, using full erosion gives zero point nine six five and zero point nine six four.

What RSS offers is a separation. It identifies sparse representation changes that genuinely recur, and it quantifies how much they contribute to a downstream readout, without equating a change in representation with a failure. A small support of around twenty features comes close to the detection that full erosion achieves.