HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

HLA. Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing. By Zhuokun Chen and five others, from Monash University and Zhejiang University. Published on the fifth of October. No code repository is linked to it.

Linear attention makes long-context autoregressive decoding efficient by compressing the history into recurrent states. That compression has a price. Once everything is folded into a state, selectively reaching back for sparse and distant information becomes difficult. Chunk-based extensions have tried to help by increasing memory capacity, but they leave a second limitation in place. The coefficients that mix one chunk with another are learned, and they may stay fixed with respect to the input content. A fixed mixture cannot adapt how history is accessed to the particular query that is asking.

Hybrid Linear Attention, or HLA, makes that access query-dependent. It is a chunk-level attention mechanism built for Gated DeltaNet, or GDN. Each completed chunk is represented as an exact affine state transition, and routing gates are computed from the content, so that which pieces of history matter is decided per query rather than once for all queries.

The gates come from compact representatives of each chunk, pooled self-attentively. Every gate then interpolates between the chunk's historical transition and the identity map. Pushed one way, the chunk contributes as it was recorded. Pushed the other way, it is passed over. Because the transition is affine, one gate governs two things at once: the chunk's additive contribution to memory, and the way it transforms the states that came before it. On top of this sits a regularizer the authors call effective-support regularization, which encourages routing to concentrate on a few chunks, so that inference can be sparse.

HLA is tested two ways: adapting pretrained models, and training from scratch. For adaptation, the models are Qwen3.5 at sizes from zero point eight billion up to nine billion parameters. Across them, HLA consistently improves over native GDN and over fixed chunk mixing. The gains reach five point five seven percentage points on LongBench-V2, and three point nine seven points on RULER.

The from-scratch setting is controlled: a model of one point three billion parameters, trained for one hundred billion tokens with a four thousand token context. There HLA improves RULER performance at every length from four thousand tokens up to thirty-two thousand. The pattern in the gains is the telling part. At four thousand tokens the improvement is zero point eight three points. At thirty-two thousand tokens it grows to four point two two points. The advantage widens as evaluation moves past the context the model was trained on.

The authors read these results as showing that composing recurrent memory in a query-dependent way improves long-context modeling, and that it keeps working beyond the training context. It does so while each chunk is still summarized by a compact affine transition. The efficiency of a recurrent state is kept. What changes is that a query now gets a say in which parts of that state it draws on.