Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding

Surfing the Activation Wave. Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding. By Raghavv Goel and four others, from Qualcomm AI Research and Stanford University. Published on the ninth of October. No code repository is linked to it.

Inside a large language model, the hidden states of different decoder layers often carry complementary information. Speculative decoding methods draw on those hidden states, yet they usually rely on a small set of target-model layers picked by fixed rules based on position. A rule like that says where to look, but it says nothing about what is actually found there, and it leaves the complementary information spread across the other layers unused.

Doing better is not as simple as trying every option. Picking the best subset of layers is inherently a combinatorial problem. The number of possible sets grows so fast that evaluating them all exhaustively is infeasible, even for models of only moderate depth. Some criterion is needed that can say, without testing every combination, which layers deserve to be used.

The authors propose a principled framework for choosing several layers' hidden states at once, built on two complementary criteria. The first is informativeness, measured through matrix-entropy analysis. The second is diversity, quantified with centered kernel alignment, or CKA. A good set should score well on both: each layer should carry a lot of information, and the layers together should repeat each other as little as possible.

The work begins as analysis. It examines Shannon entropy layer by layer, the aggregate entropy when hidden states are concatenated, and how pairwise CKA behaves across depth. Together these reveal distinct geometric and informational compartments within decoder representations.

Those findings motivate selection algorithms based on entropy and CKA, which favour sets of hidden states that are both highly informative and minimally redundant. One more ingredient sharpens them. Token-entropy filtering removes hidden states tied to tokens whose predictive entropy is low, and that improves the quality of the statistics the layer selection relies on.

The test is EAGLE3-style speculative decoding. There, the selected sets outperform fixed baseline choices, including the default rule written as two, N over two, and N minus three. The analysis also gives a clearer account of why some of those defaults can stay competitive, rather than only showing that they can be beaten.

What comes out is threefold: a practical recipe for choosing hidden-state layers for speculative decoding, a picture of how information and geometry are arranged across decoder states, and a footing for extending the same kind of analysis to related applications. The two criteria divide the labour cleanly. Entropy asks whether a layer has much to say. CKA asks whether it is saying something the other chosen layers have not already said. A set that passes both tests is rich without being repetitive, and finding one no longer depends on trying every combination. Layer choice stops being a convention and becomes something measured.