The information geometry of large language models is shared, learned, and controllable

The information geometry of large language models is shared, learned, and controllable. By Dario Picozzi, from University College London and the London Centre for Nanotechnology. Published on the tenth of September. Its code repository is worked out: about one hundred source files, with a readme and a licence.

Large language models learn similar behaviours, but two things have stayed unclear: what structure they actually share, and how to change one behaviour without disturbing the others. The claim here is that a single object answers both. That object is the Fisher-Rao geometry of next-token probabilities. The argument for choosing it is a contrast. Behaviour determines this output geometry, up to symmetries that preserve the outputs. Activation geometry, by comparison, depends on the coordinates a particular model happens to use. So if models share something real, the output geometry is where it should show.

Across transformer, state-space and recurrent models, output geometries agree more strongly than activation geometries do, and that shared geometry supports transfer of semantic categories. Agreement with human word choices increases with predictive accuracy, with scale and with training, and improves again after calibration that uses the model alone. Token probabilities and read-out geometry together predict the spectrum and its effective dimension.

The geometry is also shown to be learned in a lawful way. Under controlled language assignments, it follows the language law across architectures. Statistics of the pretraining corpus predict which held-out facts are acquired, without recalibration, while randomised experiments show that deeper evidence substantially delays acquisition, in every architecture and every evidence construction tested.

Then the controllable part. The geometry prescribes local interventions that cause minimum disturbance, and predicts their relative cost. It supports reusable control: updates learned on donor prompts transfer to unseen prompts, while preserving behaviour on reference prompts better than Euclidean control does. The same geometric correction improves steering, editing, attribution, dictionary learning and fine-tuning.

The breadth claimed is across architecture families: transformer, state-space and recurrent. The fact-acquisition result is stated for every architecture and evidence construction that was tested, which is the boundary of that claim. The control results are comparisons against Euclidean control, on behaviour measured at reference prompts. And the agreement with human word choices is conditional: it grows with accuracy, scale and training rather than holding for every model.

The three words in the title map onto the three parts of the evidence. Shared, because output geometry agrees across very different model families. Learned, because it tracks training, scale and corpus statistics in predictable ways. Controllable, because respecting that geometry makes interventions travel further while breaking less.