Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals. By Pengxiang Li and five others, from the Hong Kong Polytechnic University, Independent Researcher, and seven other institutions. Published on the eighth of October.

Looped Transformers reuse the same block over and over, and in principle more iterations should mean more reasoning. The authors observe the opposite can happen. Adding loop iterations can lower reasoning accuracy. Noisy state updates overwrite intermediate deductions that were correct, and can even undo solutions that were already complete. Later iterations are then left trying to recover lost information from a representation that has already degraded. Once an error appears in an early loop, often through long-range propagation in the recurrence, later loops find it hard to correct. The extra iterations that were supposed to add reasoning are then spent trying to repair damage. That is the degradation the paper sets out to prevent, and it is why simply looping longer has not been a reliable way to buy more accuracy.

Their argument is that looped Transformers need residual connections of their own. InfiLoop is that connection, a loop-native residual that learns two things: which past computations to retain, and how much of each new update to accept.

It combines content-based weighting with a learned temporal decay, and uses the two to maintain a running summary of the recurrent states. An exact streaming recurrence keeps the persistent aggregation memory at a constant size, no matter how large the loop count grows. The result is an adaptive update. Proposals that are unreliable get suppressed, and intermediate states that are useful are preserved instead of being overwritten. The constant memory is what makes the title's scale possible. If the summary grew with every loop, running for tens of thousands of iterations would become impractical long before it became useful.

Across a broad set of reasoning tasks, an InfiLoop model with seven million parameters outperforms existing recursive architectures. It reaches ninety seven point nine percent exact accuracy on Sudoku-Extreme, and thirteen point six percent pass at two on ARC-AGI-2.

The most telling result is on Sudoku-Extreme, where InfiLoop keeps improving as test-time looping continues past twenty thousand effective steps. The authors take this as evidence that added depth translates directly into stronger reasoning.

The diagnosis is that recurrence without protection forgets. Each pass can damage what the previous passes got right. InfiLoop gives the loop a memory that decides what to keep and what to accept, and with that in place the number of iterations stops being a liability and starts being a resource, out to tens of thousands of steps.