Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

Beneath the Surface of Chains-of-Thought. A Mechanistic Interpretation of Reasoning Operations in LLMs. By Seogyeong Jeong and five others, from KAIST and NAVER AI Lab. Published on the fourth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Reasoning in a large language model unfolds through functional operations a reader can name: problem formulation, goal decomposition, deduction. Those operations are explicitly distinguished in the text a model produces. What has not been known is whether they are distinguished anywhere inside it. The claim established here is that they are. Distinct reasoning operations exhibit corresponding geometric structure in the hidden representations.

The operations turn out to be separable in held-out representations, and separability peaks in the middle layers rather than climbing steadily with depth. Two further observations sharpen that. Across layers, token-wise operation-alignment becomes more distributed over spans instead of staying concentrated on single tokens. And identical surface tokens are represented differently depending on the operation of the chunk surrounding them, which means the structure is not a property of the words themselves.

That last point is the one the authors take care over, because it is the obvious objection. If operations differ in the words they use, or in where they fall in a sequence, then apparent separability is nothing but a lexical or positional confound. The structure is verified against exactly those explanations and survives them. Attention-masking interventions add a further constraint: operation-aligned representations at chunk onset depend on the preceding reasoning context.

The conclusion is stated as a correspondence. Language models maintain a representational relationship between the linguistic expressions of reasoning and their internal geometric structures. The names we give the steps of a chain of thought are not only descriptions imposed from outside. Something in the middle layers is keeping track of them.

Beneath the Surface of Chains-of-Thought. A Mechanistic Interpretation of Reasoning Operations in LLMs. By Seogyeong Jeong and five others, from KAIST and NAVER AI Lab. Published on the fourth of September. This is the long version: the whole paper rather than the abstract.

Reasoning models are increasingly trained on the reasoning trajectory itself, not only on whether the final answer is right. Reinforcement learning and search methods optimise the trace. That makes a question urgent which used to be merely interesting: what structure are those training objectives actually shaping inside the model? You can read a chain of thought and see it move between recognisably different activities, formulating the problem, breaking it into parts, recalling a formula, doing arithmetic. Those distinctions are explicit in the text. Whether anything corresponds to them in the representations has not been known.

To ask the question you need a vocabulary for the operations, and the authors borrow one that predates the field entirely: Polya's four-stage account of problem solving from How to Solve It. Under understanding the problem sit extraction, which identifies what the problem explicitly gives you, and direct mapping, which turns a natural-language statement into a formal one. Planning contributes decomposition, the breaking of a problem into subproblems. Carrying out the plan contributes recall of formulas and definitions, deduction from known premises, algebraic manipulation, and arithmetic computation. Looking back contributes the final answer. Eight operations, used as span labels over generated traces.

The trace is then treated not as a homogeneous token sequence but as a sequence of chunks, each a contiguous span serving one functional role. For each operation, a probe learns a direction in representation space, and the test is whether held-out spans of that operation separate from the rest along it. They do, across three models spanning two families, and the separability is strongest in the middle layers rather than rising steadily with depth. The visualisations show what that looks like: project held-out spans onto an operation's own direction and the spans carrying that operation organise along it. The datasets are chosen to cover two different demands: one is computation-heavy mathematics, the other requires applying domain knowledge to theorem-driven questions, so the operations being labelled are not all of one kind.

Two further results describe how the structure develops. Across layers, the operation signal becomes more distributed over the span: early-layer separability is sparse and token-local, while in the middle and later layers it is spread across the tokens of the chunk. So the representation of an operation starts as a property of particular tokens and becomes a property of the passage. And attention-masking interventions show that operation-aligned representations at the onset of a chunk depend on the preceding reasoning context. Mask what came before, and the beginning of the next operation is represented differently. An operation, on this evidence, is not a label the model attaches to a passage after the fact. It is something the preceding trace helps put in place before the passage begins.

The authors are careful about what the taxonomy is. They treat it as an operational framework for analysing textually expressed reasoning functions, explicitly not as a definitive cognitive model and not as an exhaustive account of the model's latent computation. The eight operations are the frequent ones drawn from a broader taxonomy. Everything is measured on mathematical and theorem-driven reasoning, on three models. And one finding complicates the tidy picture: the operation geometry persists but weakens when the reasoning contains factual errors, so the correspondence between what a trace says it is doing and what the representations encode is not unconditional.

What the work establishes is a correspondence, and correspondence is the right word for it. The names we give the steps of a chain of thought are not only descriptions imposed from outside by readers who like taxonomies. Something in the middle layers tracks them, distinguishes them across problems and contexts, and is shaped by what came before. That matters most for where the field is going: if reasoning traces are becoming the object of optimisation, then knowing they have internal geometric structure is the groundwork for intervening in that structure directly.