Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

Equal Ranking Quality, Different Decisions. Measuring and Reducing Order Dependence in LLM Scorers. By Markus Frohmann and three others, from Thomson Reuters Labs, the University of Toronto, and one other institution. Published on the twenty-sixth of September. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.

Large language models are often asked to score several candidates at once. In passage reranking, in response ranking, and in multi-document question answering, a single prompt can hold several documents or responses, and each one still comes back with its own score. Scorers like these are chosen on ranking quality. Yet what their scores actually feed is a decision: which candidates a score threshold retains, what a reader answers, or which chosen and rejected pair goes into preference training. Because the candidates share one prompt, reordering them changes their scores. The same query over the same candidates ought to yield the same decision whatever the order, and the claim here is that it often does not. Equal ranking quality does not imply equal decisions.

The sharpest evidence comes from passage reranking. Five trained scorers sit within zero point zero one of each other on nDCG at ten, so on the usual measure they are close to interchangeable. Reorder the candidates, though, and the sets each one retains overlap by only zero point six six to zero point eight four. The authors then try changing the prompt, and none of the prompt-time changes they test resolves the dependence. The only one that improves ranking quality does not measurably improve decision stability, which is the whole claim again in miniature: better ranking, no steadier decisions.

What does help is moving the fix into the weights. Order-consistency SFT, or OC-SFT, penalizes disagreement between the scores one candidate receives across different orderings. It holds ranking quality, and it leads every decision-stability measure among trained scorers on all three tasks. It is also more stable, across twelve base models, than order-averaged distillation, which trains on labels averaged across permutations. One comparison makes the size of the effect concrete. A single OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations do.

The claim concerns scorers that see several candidates in one prompt, where sharing that prompt is what lets order matter at all. It is shown across three settings: passage reranking, response ranking and multi-document question answering. OC-SFT attenuates the dependence in the weights, after prompt changes alone fell short. Nor is the argument that ranking quality is useless. It is that ranking quality by itself does not tell two scorers apart on the thing they are actually used for.

A recommendation follows directly. When scorers of this kind are compared, the comparison should report what a threshold retains and what a reader answers, not ranking quality alone. Two scorers can tie on ranking quality and still hand a downstream system different documents, depending on nothing more than the order the candidates arrived in. The code is publicly available.