Do LLMs Understand Sequential Structure? A Controlled Study of Inference and Generation

Do LLMs Understand Sequential Structure? A Controlled Study of Inference and Generation. By Jerry Wang and five others, from the University of Illinois Urbana-Champaign, Stony Brook University, and one other institution. Published on the fourth of October. No code repository is linked to it.

Language models are increasingly used as agents and simulators of behaviour. The question here is whether they recover the hidden sequential structure behind a series of actions, or only match how often each action occurs. For simulation that difference matters, because real actions are often shaped by what came before. The answer reported is that they largely do not recover it, and that behaviour which looks faithful can hide the wrong generative mechanism.

The authors use two controlled settings: two-player Rock Paper Scissors interactions, and a one-player task continuing a stochastic n-gram sequence. They test whether models can identify latent strategies, follow simple Markov rules, and sustain higher-order conditional dependencies. Their framework separates matching a distribution from following a conditional rule. Three results follow. Longer context does not improve identification. Recognising a strategy correctly does not ensure simulating it faithfully. And higher-order dependencies substantially degrade rule recovery.

These are deliberately simple, controlled games, chosen so that the true structure is known and the model's behaviour can be checked against it. The claim is about what models do with sequential structure in those settings. It does not say how they behave in richer simulations, but it is a warning about inferring mechanism from surface fidelity anywhere.

A simulator can get the frequencies right and the reasons wrong. Matching how often actions occur is the easy part; following the rule that produced them is the hard part, and the harder it becomes the more the rule slips. Separating the two is what lets that gap show.