Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure
Shuyang Xiang
Published September 20, 2026 · on the feed September 28, 2026 · ▲ 6 · arXiv 2609.23551
The abstract
Standard positional encodings represent position as a one-dimensional reading-order coordinate, but reading order alone does not determine hierarchical textual structure. We use a hierarchical rotary positional encoding (hRoPE) that represents paragraph, sentence, and token indices as separate channels, hold the token sequence fixed, intervene on the paragraph coordinate p1, and measure cross-paragraph attention with a token-distance-exact estimator. Attention is compressed relative to a token-distance-matched baseline in every corpus, but compression alone is not diagnostic of true structure: an architecturally identical channel with density-matched random labels is compressed too, more shallowly. What distinguishes real structure is the depth of compression, which is greater and corpus-dependent while the control's is not. Comparing eight corpus-only quantities across three constructs (lexical persistence, paragraph length, embedding-based coherence), none fully reproduces the cross-corpus ordering of depth, though embedding-based coherence comes closest. Compression depth, not its location, is the reproducible signature of genuine paragraph structure in our setting.
The Brief
2:06A short podcast summary of the paper. Written and voiced by AI.
Transcript
Paragraph Boundaries Are Not White Space. Compression Depth as the Signature of Hierarchical Structure. By Shuyang Xiang. Published on the twentieth of September. Its code repository is public and real: about thirty source files.
Standard positional encodings treat position as one coordinate: where a token falls in reading order. Reading order alone, though, does not determine the hierarchical structure of a text, the way it divides into paragraphs and sentences. This paper's claim is about how to tell when a model is genuinely registering that structure. Its answer is that compression of attention across paragraphs is not enough by itself. What marks real paragraph structure is how deep that compression goes.
The instrument is a hierarchical rotary positional encoding, hRoPE, which gives paragraph, sentence and token indices separate channels. The token sequence is held fixed. Only the paragraph coordinate is intervened on, and cross-paragraph attention is measured with an estimator that is exact in token distance. In every corpus, attention is compressed relative to a baseline matched for token distance.
Then comes the control that gives the result its shape. An architecturally identical channel, fed random labels matched in density, is compressed as well, only more shallowly. So compression appears with or without real structure. The difference is depth. With real structure the compression is deeper, and its depth depends on the corpus. The control's depth does not.
The author then asks what in a corpus predicts that depth. Eight quantities computed from the corpora alone, spanning three constructs, lexical persistence, paragraph length, and embedding-based coherence, are compared against the cross-corpus ordering of depth. None of them fully reproduces it, though embedding-based coherence comes closest. The conclusion is also stated with a boundary: compression depth, not its location, is the reproducible signature of genuine paragraph structure in this particular setting.
The lesson is a caution about controls. A signal that shows up for random labels too cannot, on its own, be read as evidence of structure, and here it is the magnitude, not the mere presence, that separates the two.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Paragraph breaks look like white space. Do models see more?
You want to know whether a language model registers paragraph structure. But standard positional encodings give each token one number: where it falls in reading order.
The road ahead
- 1 · The problem ~6 min Reading order gives each token one number, so it cannot show paragraphs, and distance alone already lowers attention.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Change only a paragraph label with the words fixed; attention drops, but real structure is shown by how deep the drop goes.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Give paragraphs their own channel, move them with words fixed, measure against exact-distance pairs, compare with random labels, then test corpus statistics.
- 6 · The experiment ~6 min You’ll guess: Did a channel with random paragraph labels, matched in density, also show compressed attention across paragraphs?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.