Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics

Lexical Reference Frames. Interpreting LM Residual Streams in Native Semantics. By Jian Gu and three others, from Monash University, Chongqing University, and one other institution. Published on the fifteenth of October. Its code repository is public and real: ten source files.

A language model's vocabulary is its native semantics. The embedding and the unembedding fix what a state means at the two ends of the residual stream. The trouble is everything in between. Existing methods for reading the middle take one of two routes. Either they reuse a fixed lens, or they treat the hidden states as a cloud of points. Neither reads a state the way the model itself reads it. A lens, as the authors frame it, has fixed directions at each layer, the same whatever the context. The model's own reading, they will argue, is neither fixed nor indifferent to context, and that mismatch is where the paper begins.

The authors argue that every hidden state has a native reference frame. That frame is induced by the layers that will read the state next, and it adapts to the context. On this view, what a state means is settled by what the rest of the model does with it, and a fixed set of directions cannot capture that.

Lexical reference frames approximate the native frame. They read a state through the model's own remaining layers, with a stated number of rounds of attention between positions, and a typical attention output standing in for the rest. Nothing is fitted to the model's output. The first-order version of this frame is called LexFrame one, and it is the one the main results report. The design keeps the reading honest in a specific sense. Because no fitting to the output is involved, agreement with the output later on is evidence about the frame, not something it was trained to produce.

The comparison starts from a ceiling. No lens with fixed directions per layer can exceed the square root of the native frame's coherence. At mid-depth, that ceiling comes to zero point three zero for SmolLM3-3B, and zero point three five for OLMo3-7B. The first-order frame reaches zero point nine seven on both models, and reads a state to within zero point one six nats of the model's output. The gap is the point: a fixed lens is capped well below what the model's own frame achieves at the same depth.

Three uses follow from the frame. Its coefficients attribute predictions more faithfully than attribution patching does. Its lexical basis can steer a prediction while sparing other tokens. And its confident readings would let an early exit skip zero point four five of the layers in the first model and zero point four two in the second, where lenses skip almost none.

The claim, in the end, is about access. Read through the model's own remaining layers rather than through a fixed lens, every hidden state becomes readable, attributable and steerable, in the vocabulary the model already speaks.