Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

Jev thinks "I don't know'', but doesn't say it. Introducing Sys1Cal-v1 Dataset for Probability Calibration. By Riccardo Porcedda, from little g dot A I, the Sant'Anna School of Advanced Studies, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about sixty-five source files, with a readme.

Jev, the paper says, marked the start of an era of System One Models: foundation models that return structured decisions with probability distributions attached, rather than text. Beyond low cost and high speed, Jev's central promise is that those probabilities are calibrated. No public test backs that claim. The external benchmarks that exist evaluate confidence calibration, which is a different question from whether every option probability a model returns carries the right numerical meaning. When a model puts a number on an option, does that number mean what it says? Nothing available checked that, option by option.

Sys1Cal-v1 is built to check it. The benchmark measures how far a model's returned probability distribution sits from the true one. The distance used is total variation distance from the ground-truth distribution, and from it one can estimate a soft accuracy for System One Models. The scoring does not ask whether the model picked the right answer. It asks whether the probabilities it attached to each answer were the right numbers, which is the exact promise the gap left untested.

The dataset is made of True or False questions about a proposition, call it A, for which the exact probability of A is known by construction. That is what makes a ground truth possible at all. For most questions nobody knows the correct probability, only the correct answer. Here the construction supplies the probability itself, so a model's stated number can be compared against a known one. Each item is put to the model through the three Jev primitives, named Noul, Choice and Score, so the same question is asked in each of the forms the system offers.

To show the dataset at work, the author evaluates Jev and SemIf, an open-source baseline in the Choice style. Most of the attention then goes to Jev, studying how well calibrated its Score and Choice answers are. One behaviour stands out as peculiar, and it can be explained by a single assumption: that Jev suppresses a third truth value, beyond True and False.

In Choice answers, the probability of A and the probability of not A are presented as though they add up to one. On this account there is a missing term, the probability of a third outcome, U, which is not zero but has been left out of the sum. Recovering that term lifts the median soft accuracy of Choice answers from zero point seven seven one to zero point nine seven eight. Accounting for the hidden option, in other words, brings the median close to a perfect score.

The title compresses the result into a sentence. Even when asked a strictly binary question, Jev behaves as if it wants a third option, I don't know, but the answer it returns leaves that option out of the sum. A dataset with known answers is what made that missing mass measurable.