What Makes Recurrence Effective in Looped Language Models?

What Makes Recurrence Effective in Looped Language Models?. By Xinlin Zhuang and three others, from the Chinese University of Hong Kong and MBZUAI. Published on the twenty-ninth of September. No code repository is linked to it.

Looped language models gain computational depth by sharing parameters. The same layers run again and again, so inference computation can scale without adding parameters. What has stayed unclear is when extra recurrence actually helps, and how architectural choices shape its effect. Through controlled experiments, the authors pose three questions. When does recurrence help? Where should it be applied? And how does the way it is conditioned affect performance? Their answers add up to one claim: recurrence is not uniformly beneficial, effective depth alone does not predict behaviour, and the conventional way of conditioning the loop is a weak point.

The evaluation spans inference budgets below, within and beyond the training horizon, on knowledge tasks and on reasoning tasks. On the first question, recurrence can improve reasoning beyond the training horizon while degrading knowledge performance. Harder reasoning instances, however, do not consistently benefit more.

On placement, performance depends on how distinct layers and recurrent iterations are allocated, which is why effective depth alone is insufficient to predict behaviour. Output layers that are not recurrent improve robustness to under-unrolling, and the preferred placement of input and output layers shifts with the inference budget. On conditioning, conventional initial-state injection offers limited robustness when recurrence depth varies. The authors propose history-state injection as an alternative. A channel-wise version, combined with timestep conditioning, proves a low-cost and more effective design: it better preserves knowledge under extended unrolling and improves robustness across inference budgets.

These are results from controlled experiments on looped language models, measured on knowledge and reasoning tasks. The benefit of more recurrence is tied to reasoning and to budgets past the training horizon, and knowledge performance can move the opposite way. Even within reasoning, difficulty does not reliably predict who gains. The placement preferences are not fixed either, since they vary with the inference budget, so any guidance drawn from them is conditional on how much unrolling a deployment expects.

More loops is not a single dial that turns quality up. Whether it helps depends on the task, on which layers sit inside and outside the loop, and on how the loop's state is fed back in. The authors offer their results as practical guidelines for designing looped models across variable inference budgets, with a clearer map of where recurrence helps and where it fails.