When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMs
Jiayi Li, Ruizhe Li
University of the Chinese Academy of Sciences · University of Birmingham
Published September 28, 2026 · on the feed October 2, 2026 · ▲ 6 · arXiv 2609.36138
The abstract
Before invoking external tools, an agentic LLM must select among a K-way action space: executing a call, seeking clarification, answering directly, or declining. While internal activation steering can alter these pre-execution decisions, conventional aggregate metrics obscure where altered states land and what collateral damage they inflict. We present SAKIKO, an auditing framework that formalizes representation repair via directional error discovery, router-conditioned intervention, destination-resolved verification, and prospectively frozen statistical licensing. Across seven LLMs on When2Call and MetaTool, channel-keyed interventions induce direction-specific net gains in five models; across three sealed evaluations, none of 59 budget-matched random directions matches calibrated target gain. Crucially, destination auditing shows that behavioral movement does not equal repair: an intervention achieving +55 net gain corrupts over half of the baseline-correct decisions it touches, and promising point estimates on Qwen3-4B and Gemma-2-9B are formally declined due to finite-sample uncertainty. SAKIKO establishes the necessity of outcome-resolved adjudication before claiming internal repair. Code: https://github.com/ruizheliUOA/mechanistic-tool-use-llm.
The Brief
2:14A short podcast summary of the paper. Written and voiced by AI.
Transcript
When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMs. By Jiayi Li and one other, from the University of the Chinese Academy of Sciences and the University of Birmingham. Published on the twenty-eighth of September. Its code repository is worked out: about ninety-five source files, with a readme and a licence.
Before an agentic LLM touches an external tool, it makes a choice among K possible actions: execute a call, ask for clarification, answer directly, or decline. Internal activation steering can change those pre-execution decisions. The claim is that changing a decision is not the same as repairing it, and that aggregate metrics hide the difference. They do not show where an altered state lands, or what collateral damage it does along the way.
To see that, the authors build SAKIKO, an auditing framework with four parts: directional error discovery, router-conditioned intervention, destination-resolved verification, and statistical licensing that is frozen in advance. Across seven LLMs on When2Call and MetaTool, channel-keyed interventions produce direction-specific net gains in five models. Across three sealed evaluations, none of fifty-nine budget-matched random directions matches the calibrated target gain. So the steering is doing something specific.
Destination auditing is where the picture changes. One intervention achieves a net gain of fifty-five, and yet it corrupts more than half of the baseline-correct decisions it touches. Behavioral movement, in other words, is not repair. Separately, promising point estimates on Qwen3-4B and Gemma-2-9B are formally declined, because finite-sample uncertainty is too large to license them.
The setting is tool-use decisions taken before execution, on two benchmarks and seven models, with gains appearing in five of them. The corruption figure belongs to one intervention, the one with the net gain of fifty-five. It describes that case, not a rate measured across every intervention.
A net gain can be built from fixing some decisions while breaking others, and a headline number will not say which. SAKIKO's position is that outcome-resolved adjudication has to come first: check where each changed decision actually lands before calling any internal intervention a repair.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A score went up. That is where the trouble starts in this quest.
You study an assistant that, before using any external tool, must pick one of four moves: run the tool, ask for clarification, answer directly, or decline. You can nudge its internal activations to change that pick.
The road ahead
- 1 · The problem ~7 min Nudging an assistant's internal state can raise its score while sending some decisions to new wrong moves and breaking ones that were already right.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min A nudge only counts as repair if decisions land on the right move, correct ones stay intact, and frozen statistics back the claim.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The audit finds directed errors, pushes only when a router fires, checks each decision's landing spot, then licenses claims with frozen statistics.
- 6 · The experiment ~6 min You’ll guess: One steering intervention produced a large net gain. What share of the already-correct decisions it touched did it corrupt?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.