More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models

More Choices, Fewer Decisions. Ordinal-Scale Bias in JEV-like Direct-Decision Models. By Tianxiang Gao and four others, from Jilin University. Published on the thirtieth of September. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

Direct-decision models turn text straight into structured labels and scores with low latency, which makes them attractive for classification and for automatic evaluation. The authors argue that reliability takes more than accuracy. A model also has to use the ordinal scale the user gives it faithfully. What they find is that these models compress the scale: their final decisions crowd into fewer points than the scale offers. They call this ordinal scale-utilization bias.

The models studied are JEV one point one three and three open KEV models. The investigation starts on ANLI. There JEV assigns thirty-eight point eight percent of all its predictions, and fifty-one point three percent of its errors, to the Neutral label. That happens despite an accuracy of seventy-four point nine five percent, gold labels that are nearly balanced, and candidate positions that are balanced too.

Widening the view to thirty-six ordinal datasets, final decisions use only sixty-seven to seventy-six percent of the effective gold support. On four nominal tasks, the same figure is eighty-seven to one hundred and two percent. Randomizing the order of candidates weakens the compression but does not remove it.

A controlled test then holds the items and source scores fixed, balances gold support and positions, and refines the scale from two points up to fourteen. Utilization falls for every model, reaching twenty-six to seventy-five percent at fourteen points, even though for most models the candidate probabilities stay broad. Finally, targeted BA-LoRA post-training on eight supervised scales raises gold-relative utilization from about forty-seven percent to eighty-six percent, at both KEV sizes.

The authors define the bias by what it is not. It is compression of the candidate space at the decision stage, and it is distinct from accuracy, from imbalance in the gold labels, from fixed position effects, and from the count of candidates alone. The post-training result shapes the scope too: because the compression could be trained away, the authors read it as learned and modifiable rather than a fixed limit of the architecture. The evidence is drawn from JEV and three KEV models.

The title compresses the finding into four words. Offer a model more choices, and it makes fewer distinct decisions. A high accuracy score can sit on top of that without revealing it, which is why the authors treat use of the scale as something to measure in its own right.