Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

Generalization Is Stability, Not Accuracy. Multi-Axis Evaluation of LLMs. By Nagham Omar and five others, from the Technion, Israel Institute of Technology. Published on the first of October. No code repository is linked to it.

The authors start from a definition. Generalisation in a large language model is the ability to produce consistent and semantically stable outputs when the same input is expressed in different ways. Existing work typically evaluates it through aggregate accuracy on a single prompt format, a single task, or a single set of variations. That conflates robustness with overall benchmark performance. And a single score can be improved through narrow training, or in other ways that obscure whether the model generalises at all. Two models with the same accuracy can differ sharply in how they behave when a question is rephrased, and an aggregate score has no way to show the difference.

Accuracy is not what this framework measures. The Stability-Aware Generalization Objective, SAGO, measures how much model behaviour changes for the same input under different variations and across different benchmarks. Evaluation happens at the level of individual examples, across multiple input variants, and across different aspects of behaviour. The focus is variability, not a number to push upward.

Variability is captured along several dimensions. Generation consistency is one. Internal activations are another. Confidence is a third, and response mirroring a fourth. Each dimension is a separate axis on which the same input, phrased differently, can produce different behaviour. Some of these are visible from outside, in what the model says and how sure it sounds. One, the internal activations, looks inside the model. Measuring all of them on the same examples is what lets the axes be compared with one another.

Many commonly used models exhibit generalisation instability that is statistically significant and consistent. No model generalises uniformly. The behavioural axes capture independent failure modes, so stability on one axis does not vouch for another. And cross-dataset variation can reverse model rankings.

That last result reframes the leaderboard. If moving to another dataset can flip which model ranks first, a ranking says something about the dataset as well as the models. The title's claim follows: what the authors want to call generalisation is stability, and accuracy alone does not reveal it.