Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers

Learning Chaos Without Seeing Chaos. Extrapolation of Global Dynamics in Autoregressive Transformers. By Yilun Liu and seven others, from Ludwig Maximilian University of Munich, the Technical University of Munich, and one other institution. Published on the thirtieth of September. No code repository is linked to it.

An autoregressive model learns to predict a system one step at a time, and running it recursively lets those learned dynamics unfold over long horizons. This paper asks how much of a system's broader organization such a model can recover when training only ever showed it a part of that system. Its answer is that a surprisingly narrow window may be enough. Small transformers trained on restricted parameter regimes go on to reproduce period-doubling cascades, chaotic dynamics, and attractor structures at parameters far outside anything they were trained on.

The setup is deliberately small. Transformers are trained from scratch on trajectories from several non-linear dynamical systems, including the logistic map, the sine map, the Lorenz system, and the generalized Hopf system. Each is sampled only from a restricted range of its parameters. Both the control parameters and the state trajectories are represented as sequences of continuous tokens. The models are then evaluated in closed loop, feeding on their own output, at parameter values far beyond the training distribution.

Under that evaluation, the models recover self-similar period-doubling cascades, chaos, and attractor structure, with what the authors describe as remarkable visual and numerical fidelity. The sharpest number comes from the logistic map. A transformer there reproduces successive period doublings all the way up to period one hundred and twenty-eight. The scaling ratio it yields at that finite order is four point six six eight seven, which matches the Feigenbaum constant to within five ten-thousandths.

The authors also follow how these structures emerge as training proceeds. With causal interventions, they trace how information about the control parameter is processed through attention into state prediction, and how it shapes the closed-loop dynamics that result.

The models in question are small autoregressive transformers trained from scratch, and the systems are several non-linear ones, among them two maps, the Lorenz system, and the generalized Hopf system. The phrasing of the conclusion is careful: the results suggest that a narrow window into local behavior may suffice. They do not establish it for every system or every model. The Feigenbaum match is reported for the logistic map specifically.

Learning chaos without seeing chaos is the title's own summary, and it holds up as a description of the experiment. The models were shown orderly corners of each system, and when pushed past the edge of what they had seen, they produced the cascade into disorder that the full system contains.