Graph Machine: Towards Better Pretraining via Edges
Lintai Hou
Published September 2, 2026 · on the feed September 9, 2026 · ▲ 6 · arXiv 2609.02881
The abstract
We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1). Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
The Brief
1:39A short podcast summary of the paper. Written and voiced by AI.
Transcript
Graph Machine. Towards Better Pretraining via Edges. By Lintai Hou. Published on the second of September. Its code repository is public but slight: eight source files.
Existing approaches to sparse access over a model's state make one of two compromises. Some keep a state of fixed size, so the amount the model can hold stays constant however long the input grows. Others route sparsely, but the routing is static rather than dynamic.
The Graph Machine, or GM, declines both. It maintains a state whose size is of order n, growing with the input, and reaches into that state through routing that is sparse and dynamic. Its sparse layers keep order n complexity, yet the state they can potentially access is not restricted to a constant size. The device that makes this possible is the edge.
An edge is a pointer-like object, updated differentiably by a referral mechanism that resembles pointer chasing. To test the design at pretraining scale, the authors took Qwen3-0.6B, replaced seventy-five percent of its dense Transformer layers with GM sparse layers, and pretrained the result from scratch on fifteen point seven billion tokens.
The sparse layers were run with very little retrieval. With only two tokens retrieved per KV head, out of four thousand and ninety-six, in each sparse layer, loss degrades only slightly. With four tokens retrieved, the best model marginally improves on loss.
Those numbers carry the argument. Three quarters of the dense layers are replaced, each sparse layer reads a tiny handful of tokens per head, and the loss barely moves, or moves slightly in the right direction. The claim is that a growing state and sparse, dynamic access to it can coexist, with edges as the means of access.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every sequence model trades memory against reach. Here is the wall.
You are designing a language model layer. A recurrent model keeps a fixed-size state, so it must compress its whole history into it.
The road ahead
- 1 · The problem ~7 min Every way a language model layer remembers the past gives something up: it compresses the history, reads all of it, or reads a fixed few.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Each token stores edges, weighted addresses of a few past positions; attention reads only there, and referral improves the edges layer by layer.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Start from Qwen3-0.6B, swap three in four dense layers for GM sparse layers, pretrain from scratch, and let each head read only a few tokens.
- 6 · The experiment ~6 min You’ll guess: With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, how did loss compare with the dense Qwen3-0.6B?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.