Encoded Early, Used Late: Where Transformers Begin to Act on an Inferred Partner's Expertise
Mika Okamoto, Gabriele Sarti
Georgia Institute of Technology · Northeastern University
Published September 7, 2026 · on the feed September 9, 2026 · ▲ 15 · arXiv 2609.07139
The abstract
A transformer can make an attribute linearly decodable in its residual stream at a depth where that attribute does not yet influence the output. This gap between where information is readable and where it is used has been shown for attributes stated directly in the input. We ask whether it also holds for an attribute the model must infer gradually over a conversation, namely how expert its dialogue partner is. Using ExpertCollab, a corpus of multi-turn research-planning dialogues between model-played personas at four expertise levels, we find that partner expertise is most decodable in the early layers and falls to near chance before the midpoint of the network. Counterfactual patching shows that injecting the expertise difference at the layer of peak decodability barely changes a fixed late-layer readout, whereas the same difference injected past the midpoint propagates almost completely, a separation of more than an order of magnitude. A content-matched random control and a probe-free diagnostic place the transition at the same early layer, and a statically specified control attribute stays decodable throughout. An inferred relational attribute is therefore represented well before it becomes causally active, which bounds where any attempt to read out or steer partner-conditioned behavior must intervene. We use one model on a synthetic corpus as an initial demonstration.
The Brief
2:21A short podcast summary of the paper. Written and voiced by AI.
Transcript
Encoded Early, Used Late. Where Transformers Begin to Act on an Inferred Partner's Expertise. By Mika Okamoto and one other, from the Georgia Institute of Technology and Northeastern University. Published on the seventh of September. No code repository is linked to it.
A transformer can make an attribute linearly decodable in its residual stream at a depth where that attribute has no influence yet on the output. The information is readable there, but not used there. This gap between reading and use had been shown for attributes stated directly in the input. The question here is whether it also holds for an attribute the model has to infer gradually over a conversation: how expert its dialogue partner is. The answer is yes. An inferred relational attribute is represented well before it becomes causally active.
The data is ExpertCollab, a corpus of multi-turn research-planning dialogues between personas played by a model, set at four levels of expertise. Partner expertise is most decodable in the early layers, and its decodability falls to near chance before the midpoint of the network. So the place where the attribute is easiest to read is early, and it fades from view well before the output.
Counterfactual patching tests where the attribute is used. Injecting the difference in expertise at the layer where it is most decodable barely changes a fixed readout in a late layer. The same difference injected past the midpoint propagates almost completely. The separation between those two effects is more than an order of magnitude.
Two checks back this up. A content-matched random control and a diagnostic that does not rely on probes both place the transition at the same early layer. And a control attribute that is specified statically, rather than inferred, stays decodable throughout the network, which shows the fading is particular to the inferred attribute rather than a general property of these layers.
The authors present this as an initial demonstration: one model, on a synthetic corpus. The dialogues are model-played personas rather than real people, and the finding concerns one inferred attribute, partner expertise. Whether the same separation holds for other models, other inferred attributes, or natural conversation is not established here.
The practical consequence is a bound on where to intervene. Anyone trying to read out or steer behaviour that depends on the partner needs to act where the attribute is causally active, which is past the midpoint, not where a probe finds it most clearly. The layer that looks most informative is not the one that matters.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here, a signal that is easy to read and easy to misread.
You study a large language model mid-conversation. A simple classifier can read how expert the other speaker is from early hidden layers. It is tempting to say this is where the model uses it.
The road ahead
- 1 · The problem ~7 min A probe can read a partner's expertise from early layers, but being readable there does not show the model uses it there.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Partner expertise is easiest to read early, but it only affects the output past the network's midpoint, so steering has to happen late.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Generate persona dialogues, probe each layer for partner expertise, inject the expertise difference at a layer, and measure a fixed late readout.
- 6 · The experiment ~6 min You’ll guess: Injecting the expertise difference at the layer where it is easiest to read: how much did it change the late readout?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.