Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration

Let Confidence Change, Not the Prediction. Prediction-Preserving Repair for Post-hoc Calibration. By Daehwan Kim and two others, from Hanyang University and Hankuk University of Foreign Studies. Published on the second of September. Its code repository is public but slight: one source file.

Post-hoc calibration exists to fix a model's reported confidence, not its answers. But a multiclass calibrator can change the top-one prediction as a side effect, and accuracy will not tell you it happened. Accuracy captures only the net effect on correctness, so a calibrator that flips one prediction wrong and another right looks like it changed nothing. The rate at which predictions change is a separate quantity, and the paper measures it as the top-one prediction change rate.

The framing move is to stop asking the calibrator to preserve predictions and instead repair its output afterwards. Fitting under a preservation constraint complicates the calibrator; repairing after the fact separates the two jobs cleanly. Calibration produces a probability vector, and a subsequent step edits that vector so its own argmax is back where it started.

The method works from the original and calibrated outputs alone. It determines how much mass to assign to the original top-one class, then lets the calibrated conditional distribution allocate everything remaining across the other classes. The result is a repaired vector whose argmax recovers the original prediction by construction rather than by tuning, which is what the word exact is doing in the claim: the guarantee holds for every input, not on average. On the calibration split it coordinates those repaired masses so the calibrated outputs' mean mass on original predictions is retained wherever that is attainable. Notably it alters neither the fitted calibrator nor its direct output, fits no additional supervised map, and has no hyperparameter for a user or a validation set to tune.

Across both CIFAR datasets and ImageNet-1K it attains a zero prediction change rate by construction, and lowers mean expected calibration error, negative log likelihood and Brier score against the corresponding direct outputs on every dataset. The paired gains persist under distribution shift and across different calibration set sizes, which are the two conditions under which a post-hoc adapter would most plausibly stop helping. Improving the calibration metrics while changing no predictions at all is what makes the result interesting: those two goals had looked like a trade.

The structural claim is a separation of concerns. Exact recovery of the original decision is assigned to output repair, which removes the preservation constraint from calibrator fitting entirely, and leaves the calibrator free to do the one thing it was for.

Let Confidence Change, Not the Prediction. Prediction-Preserving Repair for Post-hoc Calibration. By Daehwan Kim and two others, from Hanyang University and Hankuk University of Foreign Studies. Published on the second of September. This is the long version: the whole paper rather than the abstract.

This paper turns on one measurement, and the measurement is the kind that reframes a field's assumptions rather than adding to them. Apply a standard calibration method, Vector Scaling, to a pretrained ResNet-50 on ImageNet, and accuracy drops by zero point two four percentage points. Almost nothing. That same operation changes the model's top prediction on five point seven four percent of examples.

To see why those two numbers can differ so much, you need the structure of the problem. Post-hoc calibration leaves a trained classifier's parameters fixed and fits a separate map to its outputs, aiming at confidence calibration: predictions reported with confidence p should be correct with frequency p. But in standard multiclass prediction a single probability vector is doing two jobs at once. Its argmax selects the predicted class, and the value at that coordinate encodes the confidence. A map that modifies the vector to fix the second thing can therefore change the first. Some calibration maps happen to retain the original top prediction; others may not.

That gap is what the paper's new measure, the top-one prediction change rate, exists to expose, and only ground-truth labels could have revealed how each revision affected correctness. The conventional response to unwanted changes is to prevent them at fit time, constraining the calibration map to preserve the original top prediction. But that narrows the class of admissible transformations, and prior work has made the tension explicit by naming accuracy preservation and high expressive power as competing desiderata: more expressive power can cost classification accuracy. So the authors ask a different question. What if the preservation constraint were removed from fitting and imposed afterward?

Their adapter repairs the fitted calibrator's full probability vector, using only the original and calibrated outputs. It determines the mass to assign to the original top class, then lets the calibrated conditional distribution allocate everything remaining across the others, producing a vector whose own argmax recovers the original prediction. On the calibration split it coordinates those repaired masses so the calibrated outputs' mean mass on original predictions is retained where attainable. Three properties define it precisely, and together they are what make it an adapter rather than another calibrator. It leaves both the fitted calibrator and its direct output unchanged. It fits no additional supervised map. And it introduces no hyperparameter for a user or a validation set to tune.

Across CIFAR and ImageNet, the adapter attains a zero prediction change rate by construction, and lowers mean expected calibration error, negative log likelihood and Brier score relative to the corresponding direct outputs on every dataset. Getting all three calibration metrics to improve while changing no predictions at all is the combination that makes the result interesting, because those two goals had looked like a trade: the whole reason preservation was imposed at fit time was the belief that you had to give up expressive power to get it. The paired gains persist under distribution shift and across calibration-set sizes, which are the two conditions where a post-hoc adapter would most plausibly stop helping. Shift matters because the repair is constructed from the calibration split, and a smaller calibration set matters because the coordination step is the one part of the method that looks at data at all.

There is no limitations section, so the boundaries are the ones implied by the setup. Everything is image classification with standard architectures, and the measurement that motivates the work is demonstrated on one calibrator and one model before being generalised. The method also inherits whatever the underlying calibrator does to confidence: it repairs the decision, not the calibration, so a badly fitted calibrator repaired exactly is still badly fitted. And the case for preserving the original prediction is assumed rather than argued. Sometimes the calibrated model's revised choice is the better one, and this method gives that up by design.

The structural contribution is a separation of concerns, and it is cleaner than the compromise it replaces. Fitting no longer carries a constraint that limited what a calibrator could express, and exact recovery of the original decision becomes the job of a subsequent repair step. The measurement is what makes the case, though. A field that had been reporting accuracy change was reporting the net of three effects, one of which cannot show up in it at all.