One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

One Bias Score, Two Factors. How Mitigations Affect Option Bias in Multiple-Choice Evaluation. By Hyunjong Ok and two others, from the Pohang University of Science and Technology. Published on the fourteenth of October. No code repository is linked to it.

Shuffle the options of a multiple-choice question and a model may change its answer. That effect has been studied under two names. Selection bias is a preference for certain option letters. Order sensitivity is answers changing across orderings of the options. This work connects the two through a single score, the pooled selection-rate standard deviation, or pooled SD, which measures how unevenly option letters get selected across questions and orderings.

The central result is a clean split. When every option of each question appears equally often at each position, the pooled SD is exactly the product of two factors, neither of which depends on the correct label. One is the flip rate: the fraction of questions whose chosen option changes across orderings. The other is the positional skew: the standard deviation of option-letter selection rates among those questions.

Put simply, one factor asks how often a model changes its mind when the options move. The other asks, when it does change its mind, how lopsided its pull toward particular letters is. Multiply the two, and you recover the single bias score.

Across forty-one checkpoints and twelve benchmarks, the two factors track different things. The flip rate follows accuracy and output uncertainty. The positional skew follows the estimated prior over option symbols. So the first factor behaves like a matter of how sure and how right the model is, and the second like a matter of which letters it leans toward.

The split also sorts the mitigations. Prior correction, aimed at selection bias, mainly reduces the positional skew. Aggregating each ordering with its reversal, aimed at order sensitivity, mainly reduces the flip rate. Both shrink the range of accuracy across correct-answer letters by similar amounts, but aggregation leaves fewer reversals in how models rank. Applying prior correction first and then aggregating reduces both factors, and narrows that accuracy range further.

The exact product holds under a stated condition: each option of each question appearing equally often in each position. And one mitigation's gain is not entirely what it looks like. Aggregation lowers the flip rate partly by settling questions on the same wrong option. A stable answer, in that case, is not a correct one. A model that always picks the same wrong option across every ordering never flips, and so scores as perfectly stable on that question.

That caveat drives the recommendation. Report both factors, so that the effect of a mitigation can be told apart, and report accuracy beside them, so that stability is not mistaken for correctness. One bias score turns out to hide two different behaviours, and the fixes people already use each act mostly on one of them.