Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It. By Zehao Jin and two others, from the Georgia Institute of Technology. Published on the twenty-ninth of September. Its code repository is worked out: about eighty source files, with a readme and a licence.

Pretrained transformers use little of their depth to follow references in context. Think of a chain of program lines, each pointing back to an earlier one, and a model asked to follow it. Across thirteen base models, the reliable reach is only about one point four to three point six lines, and adding extra pretrained loops adds little. The claim is that this is a default, not a ceiling: a tiny edit unlocks far more, without changing any of the model's own weights.

The edit is a rank-eight LoRA, trained on the task and placed at one early layer, with every model weight frozen. On twenty-four-line chains, Qwen3-8B goes from fifteen point five percent to ninety-nine percent exact accuracy, and a longer-trained LoRA reaches fifty lines. Ouro-1.4B reaches sixty lines after four loops, and at least one hundred and sixty after eight.

The authors also trace how. The LoRA starts a relay: program lines pass on their chain identity through a short range of middle layers, and frozen heads read progressively further up the chain. Remove the attention to the parent line and the relay stops. A measurement on the frozen model locates the last useful intervention layer within tolerance in three of four held-out models.

These are chain-following tasks with task-trained adapters. Beyond them, the abstract reports only that task-specific LoRAs also improve MuSiQue, and the layer-locating measurement worked in three held-out models out of four, not all of them.

What a model does by default and what it can be made to do with a rank-eight nudge are far apart. Default answers, on this reading, understate the computation that a very small edit can reach.