BiasReducer: Adaptive Bias Mitigation for Reward Models

BiasReducer. Adaptive Bias Mitigation for Reward Models. By Shuang Liu and four others, from Carnegie Mellon University, the Chinese University of Hong Kong, Shenzhen, and two other institutions. Published on the twenty-sixth of September. Its code repository is worked out: about fifty source files, with a readme and a licence.

Reward models score the responses of large language models, and those scores steer training toward human preferences. But a reward model can favor superficial attributes, such as a response's length or how confident it sounds. A model trained against it then learns to produce responses that score higher without being more correct. The reward goes up, and the thing the reward was meant to stand for does not.

The existing fixes come in two kinds. One retrains the reward model, which costs additional data and compute. The other applies a fixed correction to a single known bias, a preference for longer responses being the classic case. Those editing methods need the bias named in advance, and they apply the same edit to it every time. So the first option is expensive, and the second only helps with the bias someone already knew to look for, in the same amount no matter which data the reward model is facing.

BiasReducer is a lightweight framework that does neither. It edits only the linear reward head, and it chooses which edits to apply for each new dataset. The bias does not have to be known ahead of time, and the correction is not fixed.

It works in three steps. To begin, an encoder in the style of a sparse autoencoder, or SAE, learns which attributes the reward model is sensitive to: length and confidence among them. Next, BiasReducer learns how to reduce the reward model's dependence on each of those attributes, deciding which direction to adjust the reward head in, and by how much. Finally, given a new dataset, it ranks the attributes by how much they influence reward scores, selects the relevant ones, and edits the reward model to match. The first two steps build a vocabulary of possible biases and a correction for each. The third decides, dataset by dataset, which of those corrections to apply.

BiasReducer consistently makes reward models more robust to biases toward superficial attributes. Across five reward models, the variant called BiasReducer-M improves three benchmarks on average by eight point three percentage points, by eighteen points, and by six point nine points respectively. That outperforms the two training-based baselines, even though BiasReducer itself does no retraining.

The gains carry downstream as well. Models guided by the edited reward models show less unnecessary verbosity and less sycophancy, while their judged quality stays comparable.

Two questions sit inside every bias fix: which bias, and how much to correct it. Earlier editing methods answered the first by assumption and the second with a constant. BiasReducer learns both, per dataset, and it does so while touching nothing but the linear reward head.