Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment

Making LLMs Say What They Think. Measuring and Improving CoT-Interpretability Alignment. By Yihuai Hong and three others, from New York University and NYU Shanghai. Published on the thirtieth of September. Its code repository is worked out: about seventy source files, with a readme.

Chain-of-thought traces are often treated as a window onto how a large language model reaches its answer. Growing evidence says the window is unreliable. A model's chain of thought often fails to reflect its internal computation, and it can be changed without affecting the final answer. If the written reasoning and the real reasoning can come apart, then reading the trace tells you less than it seems to, and what has been missing is a direct measure of how far apart they are.

CoT-Interpretability Alignment, or CIA, is that measure. It scores the agreement between a model's chain-of-thought traces and its internal reasoning strategies, where those strategies are detected with interpretability tools. High agreement means the model says what it is actually doing. Low agreement means the trace describes one route while the computation takes another.

The metric is evaluated on three tasks: two-hop question answering, hint intervention, and integer multiplication. Each is run across three LLMs. The authors then go beyond measurement and try to improve CIA through post-training, using two reward signals together: task accuracy, and parametric faithfulness.

Alignment is limited across all three tasks, ranging from forty-four point eight to seventy-five point nine percent. Even at best, then, a sizeable share of traces do not match what the model is computing. Post-training changes that. It substantially improves the parametric faithfulness of the chain of thought, and does so while maintaining or improving task accuracy. The paper also analyses how these gains generalise.

Two contributions arrive together. One is a framework for auditing whether a chain of thought is parametrically faithful. The other is a pathway toward models whose explicit reasoning can be trusted more. Code and data are released.

The structure of the work is a loop. Interpretability tools reveal what the model is computing. The chain of thought claims what it is computing. CIA measures the agreement between those two, and that measurement then becomes part of the reward that trains the model. A number that started as a diagnosis ends up as a training signal, and the result is a model whose written reasoning stays closer to the reasoning it actually performs, with no loss in the answers it gets right.