SparseEngine: Sparse-First Inference Engine
Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu
Published September 30, 2026 · on the feed October 9, 2026 · ▲ 18 · arXiv 2609.39068
The abstract
Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.
The Brief
1:58A short podcast summary of the paper. Written and voiced by AI.
Transcript
SparseEngine. Sparse-First Inference Engine. By Jitai Hao and three others. Published on the thirtieth of September. Its code repository is worked out: about nine hundred and fifty source files, with a readme and a licence.
Language model agents that work over long contexts build up interaction histories, and those histories strain both the memory of the KV cache and the computation of attention. Sparse attention cuts these costs. But sparse methods differ in how they represent the cache and in their workflows, and that heterogeneity makes them hard to fit into existing inference engines. Earlier abstractions for serving sparse methods do not solve this either, since each supports only particular layouts or particular workflows.
SparseEngine is an inference engine built from the ground up with sparsity first, rather than added on later. At its centre is a shared lifecycle contract. Under that contract, each method keeps control of its own KV representation and its own computation, while the transitions of its state are coordinated with common serving infrastructure. The method decides what it stores and how it attends. The engine handles the rest.
The engine supports fifteen methods across four categories. Two features manage state across requests. Chain Cache lets methods that evict KV resume from retained history, instead of starting over. Prefix-Cache Pruning, which is controllable, removes KV from chosen regions of history while still preserving logical-prefix matching, so that requests can continue to share what came before.
While keeping method quality intact, SparseEngine delivers over ten times higher throughput with KV eviction. Against vLLM at matched concurrency, it decodes over two point five times faster. On agent benchmarks it gives over twice the end-to-end speed.
The design choice is to let each sparse method stay itself, and to make the engine meet it at a shared contract rather than forcing one layout on all of them. That is what sparse-first means here: the variety of methods is the starting assumption, not an inconvenience to be engineered away.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free