ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models

ReaDy. Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models. By Bohan Sun and five others, from StartLux, the University of Chinese Academy of Sciences, and one other institution. Published on the fourteenth of October. Its code repository is public but slight: nine source files.

Speculative decoding speeds up autoregressive generation by drafting several tokens per step instead of one. Feature-based drafters build those drafts from the target's hidden features. The trouble is emphasis. A hidden feature carries whatever context the target found useful for its own next token, and that may not be the context a later draft position needs. Each position further out is guessing with information shaped for somebody else.

Hybrid linear attention models, or HLA models, offer another source. They maintain compact recurrent states that accumulate historical associations as the sequence goes on. ReaDy is, to the authors' knowledge, the first speculative decoding method to use those recurrent states for position-specific context retrieval during drafting. Rather than every draft position leaning on the same hidden features, each one gets to ask the state for what it, in particular, needs.

Concretely, each draft position is given a learned query, and that query retrieves supplementary context directly from the target's committed recurrent states. A lightweight state reader then folds the retrieved context into the draft representations through residual updates. Two things keep the cost down: low-rank state reads, and fused operators, which hold the extra computation and the extra I/O overhead low.

ReaDy does not start from scratch. It builds on DSpark's parallel backbone and sequential prediction head, and adapts them with a lightweight two-stage strategy. The design choices, namely how much capacity the reader has, which states it reads from, and where the residual goes, are each examined through ablation studies.

Evaluation spans seven benchmarks covering mathematics, code generation and dialogue, using Qwen3.8-27B on a single NVIDIA B200 GPU. Against plain autoregressive decoding, ReaDy delivers an average decoding speedup of three point one six times. Against DSpark, the system it builds on, it wins on every benchmark. On average, throughput improves by seven point nine seven percent, and acceptance length grows by eight point five percent.

Hybrid linear attention models already carry a running summary of the past for their own purposes. ReaDy treats that summary as something a drafter can query, with a separate question for each position. The cost is a light reader and some low-rank reads. The return, on this model and this GPU, is a margin over DSpark that holds on all seven benchmarks.

ReaDy. Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models. By Bohan Sun and five others, from StartLux, the University of Chinese Academy of Sciences, and one other institution. Published on the fourteenth of October. This is the long version: the whole paper rather than the abstract.

Speculative decoding speeds up generation by letting a small drafter guess several tokens ahead, then having the big target model check them all in one pass. The newest drafters guess a whole block at once. DSpark, the baseline here, drafts seven positions in one forward pass, then a light sequential head links neighbouring guesses. The catch is that the backbone must prepare context for every future position at the same time, and the context it gets is the target's hidden features. Those features were learned for predicting the very next token. They reflect what the target needed at positions already seen, not what the sixth draft position needs. The alternative, reading the target's key and value cache directly, gives fresh access but costs memory traffic that grows with the prefix.

The target here is a hybrid linear attention model, Qwen three point eight. It has sixty-four layers, repeating three Gated DeltaNet layers and then one full attention layer. The full attention layers keep a growing cache of every token. The Gated DeltaNet layers do something different: each head keeps one fixed matrix, here one hundred and twenty-eight by one hundred and twenty-eight, that accumulates associations between keys and values over the whole prefix. The memory is the matrix, and a query picks which part of it you see.

That split between state and readout is the paper's theoretical starting point. Suppose you can only see the answers a fixed state gave to some known queries. A new query can be broken into a part inside the span of those known queries and a part outside it. The inside part is fully determined by the answers you already have. The outside part is not: two different states can agree on every answer you were shown and still disagree on the new one. The authors prove this, and stress that it concerns one idealized state, not the drafter's full multilayer features. Whether direct access helps is left as an empirical question.

A second identity explains why reading only the committed state still makes sense. Unroll the recurrence a few steps into the future, and each future readout splits in two: the committed state answering a transformed query, plus the effect of writes not yet made. The first part is still a question to the same committed matrix. The right query depends on a future the drafter cannot see, so ReaDy learns query directions from the draft representations.

The reader sits in the final draft layer, after attention and before the feedforward block, so each position's representation already holds the backbone's context. That representation is normalized and squeezed through a rank-eight bottleneck, then expanded into a full one hundred and twenty-eight dimensional query. Separately, the normalized state's value side is compressed to rank eight, a projection shared by every position in the block. Readouts from the selected layers and heads are joined with the log of each state's original scale, since normalization discarded it, passed through a bottleneck feedforward network, and added back as a residual. Verification is unchanged. Only states from accepted tokens are ever read; states from rejected guesses are dropped.

Cost was engineered too. The state source is fixed in size whatever the prefix length, unlike a token cache. Fused kernels avoid extra copies and intermediate tensors. Gathering falls by seventy-one point eight percent and the full reader by twenty-eight point one percent at the operator level. Training is light: stage one trains only the reader on twenty thousand samples, stage two adds low-rank updates to the drafter on fifty thousand more, and the target stays frozen.

Across seven benchmarks in maths, code and dialogue, on one NVIDIA B200 GPU with greedy decoding, ReaDy averages three point one six times the speed of plain autoregressive decoding, against two point nine three times for DSpark. That is a seven point nine seven percent throughput gain, with tokens per verification step up eight and a half percent. It wins on every benchmark.

The comparison that actually tests the thesis is a control. ReaDy's training recipe differs from DSpark's, so some gain could be just more fine-tuning. The authors fine-tune DSpark the same way without state access. That alone lifts throughput from about two hundred and eighty to about two hundred and eighty-six tokens per second. ReaDy reaches about three hundred and three, a further five point six eight percent from the state reads themselves.

The ablations say where the value lives. Layer sixty-two, near the top, is worth far more than layers two or thirty alone. Layers thirty and sixty-two together work best, and adding layer two makes things worse. Injecting at every draft layer instead of once changes acceptance not at all and only costs speed. Complete seven-token blocks survive more often too: on GSM8K, from about forty-six percent of rounds to about fifty-eight.

There is no limitations section. The evidence is narrower than the headline. One target model, one GPU, a request concurrency of one, greedy decoding only, and outputs capped at one thousand and twenty-four tokens. No error bars or repeated runs are reported. The appendix flags something easy to miss: the checkpoint has overlap between training and evaluation, so the survival curves describe decoding behaviour, not held-out generalization, and the authors say they do not isolate a semantic retrieval mechanism.

The idea to keep is that a hybrid model's recurrent state is a memory you can ask new questions of, at a cost that does not grow with the prefix. A hidden feature is an answer someone else already chose; the state is open to a query shaped by what each draft position needs. ReaDy's contribution is showing that a tiny, carefully placed reader can turn that into faster decoding, and that the gain survives a fair fine-tuning control.