Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk, Nikita Dragunov, Temurbek Rahmatullaev, Polina Druzhinina, Anton Razzhigaev, Ivan Oseledets and 1 others
Published September 24, 2026 · on the feed September 25, 2026 · ▲ 81 · arXiv 2609.29845
The abstract
While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions. We term this the Superposition Linearity Hypothesis. We provide evidence that superposition is an intrinsic property of the Transformer architecture rather than an emergent consequence of training; in fact, we observe that it tends to diminish as pretraining progresses. However, we demonstrate that linearity can be substantially restored through lightweight fine-tuning, significantly reducing the divergence between the predicted next-token distribution and the average of the individual next-token distributions. Finally, we introduce a guided decoding procedure that disentangles superposed outputs, enabling the simultaneous generation of two coherent continuations from a single forward pass.
The Brief
1:36A short podcast summary of the paper. Written and voiced by AI.
Transcript
Your Transformer Can Hold Two Thoughts at Once. Evidence of Linear Superposition in LLMs. By Pavel Tikhonov and eight others. Published on the twenty-fourth of September. No code repository is linked to it.
A large language model relies on highly non-linear components, and yet the authors report a fundamental linearity in how these models behave. Take inputs from two distinct text streams and combine them linearly. The model's output is then a superposition of the two individual next-token distributions, as if it were predicting for both streams at once. The authors name this the Superposition Linearity Hypothesis.
Two lines of evidence bear on where the property comes from. The first points to the architecture. The authors argue that superposition is intrinsic to the Transformer itself, rather than an emergent consequence of training. The second supports that reading from an unexpected direction: as pretraining goes on, the superposition tends to diminish rather than grow. Training, in other words, is not what puts it there.
That sets the limit on the claim. In a fully pretrained model the linearity is weaker than the architecture alone would give, so the hypothesis describes a tendency that pretraining wears down. What training erodes can be brought back, though. Lightweight fine-tuning substantially restores the linearity, measured as a significant drop in the divergence between the predicted next-token distribution and the average of the individual ones.
The finding then becomes a tool. A guided decoding procedure disentangles the superposed output, so that a single forward pass yields two coherent continuations at the same time. A property that looked like a curiosity of the architecture ends as a way to generate twice from one pass.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A highly non-linear machine. A surprisingly linear question.
Language models are built from highly non-linear parts, so you would expect mixing two inputs to give nonsense.
The road ahead
- 1 · The problem ~7 min Language models are full of non-linear parts, so blending two texts into one input should scramble everything, yet whether that expectation holds end to end is an open question.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Average two texts' embeddings and the model outputs a mix of both texts' next-token predictions, a property the architecture brings and pretraining wears down.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Average two texts' embeddings into one input, fine-tune the model to output the mix of its separate predictions, then decode two continuations apart with guidance.
- 6 · The experiment ~6 min You’ll guess: As pretraining progresses, what happens to a language model's tendency to output a mix of two streams' predictions?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.