DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems

DecisionPivot. A Diagnostic Benchmark for Decision Models in LLM Systems. By Xialie Zhuang and nine others, from StartLux. Published on the twentieth of October. Its code repository is worked out: about sixty source files, with a readme and a licence.

Inside LLM systems, decision models do quiet but load-bearing work. They support routing, guardrails, and the verification of outputs. They are usually judged by item accuracy, the share of individual questions answered correctly. The authors argue that this number alone cannot establish two things that matter. It cannot show whether a model makes the right decision when the input is varied in controlled ways, and it cannot show whether the confidence the model attaches to its predictions can be relied on.

DecisionPivot is a diagnostic benchmark built to measure both. Models are assessed on item accuracy, but also on contrast-group accuracy, on the ordering of their probabilities, on calibration, and on empirical selective coverage. Each of the forty decision models and fourteen general models is run under a documented protocol specific to that model.

In total the benchmark holds forty-eight tasks and forty thousand and thirty-three items, spread across seven decision roles. It has two tracks.

The core track contains seven thousand eight hundred items, drawn from thirteen generated task families. These items are organised into contrast groups, sets of closely related variations, and their answers are computed and then verified by solvers implemented independently.

The complementary reference track contains thirty-two thousand two hundred thirty-three items taken from thirty-five public test sets, and it keeps the original evaluation metric of each set.

The results show large differences that aggregate scores hide. Two models of twenty-seven billion parameters each sit only two point six points apart on the reference track. On contrast groups, one reaches seventy point one percent accuracy and the other forty-four point two percent.

A second gap separates ordering from correctness. Across two thousand four hundred binary contrast pairs, Perplexity Decider ranks the positive member above the negative one in ninety-two point three percent of pairs, but answers both members correctly in only seventy-four point one percent. Ranking two options sensibly is not the same as getting both decisions right.

Models with similar item accuracy also differ in how reliable their high-confidence predictions are. And effort matters: on a separate subset of one thousand two hundred ninety-six items, GLM-5.3 reaches ninety-three point five percent contrast-group accuracy at a higher reasoning setting, against eighty-one point five percent at its lowest effort.

A single accuracy figure, then, can make two very different decision models look alike. By tracing each error to the pivotal detail behind it and to its level of difficulty, DecisionPivot offers fine-grained diagnostics of decision failures that aggregate evaluation on its own cannot reveal.