Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention

Multi-layer Indexer. Overlapping Selection and Attention in DeepSeek Sparse Attention. By Serdarcan Dilbaz and three others, from Writer. Published on the second of November.

Sparse attention makes long-context inference cheaper by having each query attend only a subset of the sequence. That subset can come from a fixed pattern, or, in recent production LLMs, be learned per query. DeepSeek Sparse Attention, or DSA, is the attention of DeepSeek-V3.2 and of the GLM-5 family. In it, every layer is preceded by a lightning indexer, which scores the whole cache and keeps the two thousand and forty-eight highest-scoring tokens for the layer to attend. At long context, the expensive part is not the sparse attention that indexer feeds. It is the indexer scan itself, and that is where the decode time goes. The step meant to save work has become the bottleneck, and because each layer waits for its own indexer, that cost is paid in sequence, layer after layer.

The opening comes from an observation: adjacent layers select nearly the same tokens. A layer can therefore attend the selection made by the previous indexer layer, instead of waiting on its own. That removes the dependency. Each indexer can then run concurrently with the attention of its layer, rather than strictly before it. The scan does not get cheaper. It moves off the critical path, overlapping with attention that has to happen anyway, which is the overlap the title refers to.

The result is the Multi-layer Indexer, or MLI. It needs no retraining and no new weights. Its default form changes no kernel and runs inside SGLang's production decode graph, and an optional fused decode kernel accelerates it further on DeepSeek-V3.2. It also composes with IndexCache, the cross-layer reuse that GLM-5.2 ships as IndexShare. The two divide the work between them. IndexCache drops the indexer from some layers altogether, and MLI hides the scan on the layers that keep one. One removes indexer work; the other keeps what remains from stalling decode.

DeepSeek-V3.2 and GLM-5 were served under IndexCache's protocol. Retention, the fraction of layers that keep an indexer, ran from one half down to one in thirty-two, and context ran from ten thousand to one hundred and ninety-five thousand tokens. At equal retention, the composition decodes faster than IndexCache alone. At one half retention, that is up to fourteen point five percent faster on DeepSeek-V3.2 and nine point eight percent faster on GLM-5.

At one quarter retention, the gains are seven point nine percent and five point one percent. They are largest where IndexCache keeps the most indexers, and they grow with context. A stronger comparison also holds. At retention one over r, the composition beats IndexCache at one over two r, while keeping twice as many fresh selections. At one hundred and twenty-eight thousand tokens, it reaches one point three two times the unmodified model's decode throughput at one quarter retention, above the one point two eight times IndexCache reaches at one eighth. Accuracy on LongBench, RULER and LongBench v2 stays within a few percentage points of IndexCache's.

Nothing about how tokens are chosen is learned anew. What moves is when the choosing happens. Borrowing the neighbouring layer's selection turns a scan that sat in line ahead of attention into one that runs beside it.

Multi-layer Indexer. Overlapping Selection and Attention in DeepSeek Sparse Attention. By Serdarcan Dilbaz and three others, from Writer. Published on the second of November. This is the long version: the whole paper rather than the abstract.

A model like DeepSeek-V3.2 reads a long context sparsely. At every layer, for every new token, it attends only two thousand and forty-eight of the tokens in its cache. The catch is how it picks them. Before each layer's attention, a small scorer called the lightning indexer rates every cached token and keeps the best. Sparse attention costs the same whatever the context length. The indexer's scan does not: it reads every key. At long context the scan becomes the largest piece of the attention side of the layer, two thirds of it at the longest context measured, and attention waits for it at every layer. The authors, from Writer, ask whether it has to wait at all.

Two facts set up the idea. First, the selection is the only thing tying the indexer to attention. Attention needs to know where to look, but shares no computation with the indexer. Second, neighbouring layers pick similar tokens. On GovReport documents, the Jaccard overlap between consecutive layers' selections stays near zero point four two on DeepSeek-V3.2 and zero point two six on GLM-5. An existing method, IndexCache, already exploits this. It keeps the indexer on only a fraction of the layers, and the layers in between copy the nearest earlier selection. GLM-5.2 ships that idea as IndexShare. The authors frame the indexer's cost as a product: how many indexers run, and how long each one is waited for. IndexCache cuts the first. This work goes after the second.

The Multi-layer Indexer, or MLI, changes exactly one thing: which selection a layer attends. Instead of its own, a layer attends the selection made by the previous layer that ran an indexer. Its own indexer now feeds the next such layer, so nothing in the current layer waits for it. As soon as the layer input exists, the indexer is placed on a second CUDA stream, while attention, the output projection and the feed-forward block proceed on the main stream with the older selection. The side stream is joined where the new selection is first needed. The whole pattern is captured once into SGLang's decode graph, with the production kernels unchanged. Every indexer computes exactly the selection it always did; only its consumer moves. And every selection is still recomputed for the current token, unlike IndexCache, where a shared layer attends a selection never computed for it. One scan stays serial: the first indexer layer, which has nothing earlier to borrow from.

The schedule has two independent knobs: retention, the fraction of layers keeping an indexer, and lag depth, how many indexer layers back a selection comes from. Stacked on IndexCache, MLI acts on the retained layers while IndexCache's copies cover the rest. DeepSeek-V3.2 also gets a fused kernel that runs the sparse attention and the indexer scan in one launch, so the scan never competes for compute with a separate launch. Finally, an optional repair: train only the retained indexers, about half a percent of the parameters, toward what their consumer layers would have selected themselves. The identical recipe is applied to IndexCache, so distilled arms are always compared in pairs.

Both models were served on one node of eight H200 GPUs, every arm of a comparison in the same job, with the unmodified model run first and last to catch drift. With every indexer retained and a single request, MLI alone lifts decode throughput by twenty to twenty-six percent on DeepSeek-V3.2, and fourteen to seventeen percent on GLM-5. Against IndexCache at the same retention, the combination is up to fourteen point five percent faster on DeepSeek-V3.2 when half the indexers are kept, shrinking toward parity as fewer scans remain to hide.

The comparison that most directly tests the thesis is the halving pair. The combination runs faster than IndexCache at half its retention, while attending twice as many fresh selections. At the longest DeepSeek context, keeping a quarter of the indexers, it reaches one point three two times the base model's speed, against IndexCache's one point two eight with an eighth kept. On quality, the bootstrap intervals for these pairs span zero, with a single exception on RULER. And at the cliff, freshness matters: on GLM-5, IndexCache with a sixteenth of its indexers collapses, losing sixty-one percent on RULER, while the combination one halving above does not.

There is a limitations section, and it is short: the gain falls as more requests decode together, the fused kernel loses to the stream version on GLM-5, and GLM-5.1 and GLM-5.2 are not tested separately. The rest of the paper adds more. The quality results are single runs, as the authors state. The intervals resolve differences of a point or two only on RULER. Distillation is uneven: at mild retentions it is neutral to slightly negative, and only at aggressive retention does it rescue a collapse. Quality was only measured at lag depth one. And the project used roughly three times the compute behind the reported numbers, including runs voided by the authors' own checks.

The idea to carry away is that a dependency which looks structural can be only an ordering. The indexer never gave attention anything except where to look, and an earlier layer already had a good answer. Move the consumer, keep the arithmetic, and the scan runs in the shadow of work that was happening anyway. The authors push one step further. The indexer's score and attention's logit are the same kind of low-rank bilinear interaction, so the scan is a second reader of one primitive, and sparse attention models should be designed around that primitive rather than placing a selector in front of it.