Evolving Infra: Agent-Adaptive LLM Serving at Scale
Hao Kang, Weili Xu, Ziyang Li, Zheng Du, Hanchen Li, Tian Xia, Song Bian, Tushar Krishna and 3 others
Georgia Institute of Technology · University of Illinois Urbana-Champaign · University of California, Berkeley · NVIDIA · OpenAI · Massachusetts Institute of Technology
Published November 9, 2026 · on the feed November 12, 2026 · ▲ 11 · arXiv 2611.12304
The abstract
Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limitations lead to load imbalance between prefill and decode nodes and eviction of KV caches that agents will reuse. We present Evolving Infra, an agent-adaptive serving framework that adapts compute allocation to agent workloads and KV cache management to agent context management at runtime. We decouple compute allocation from key--value (KV) cache management, allowing nodes to switch efficiently between prefill and decode. Building on this design, an online planner matches the PD split to workload composition, while an agent-aware distributed KV manager guides cache eviction and transfer with signals from agent context management. Across coding, math, and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, Evolving Infra reduces mean end-to-end request latency and P95 TTFT by up to 4.1x and 14x over Dynamo, NVIDIA's production framework.
The Brief
1:54A short podcast summary of the paper. Written and voiced by AI.
Transcript
Evolving Infra. Agent-Adaptive LLM Serving at Scale. By Hao Kang and ten others, from the Georgia Institute of Technology, the University of Illinois Urbana-Champaign, and four other institutions. Published on the ninth of November.
Agentic workloads are now a major source of demand for LLM inference. Serving systems that disaggregate prefill and decode are a common way to meet it, but the authors find two weaknesses in the existing ones. They offer limited flexibility to adapt compute allocation and KV cache management at runtime, and they rely on coarse signals that only partly capture how diverse and dynamic agent workloads are. Two costs follow: load imbalance between prefill nodes and decode nodes, and eviction of KV caches that agents are going to reuse.
Evolving Infra is a serving framework that adapts to agents at runtime in two places. Compute allocation follows the agent workload, and KV cache management follows the agents' own context management. The enabling move is to decouple compute allocation from KV cache management, which lets a node switch efficiently between prefill and decode.
Two components build on that decoupling. An online planner matches the split between prefill and decode to the composition of the workload. An agent-aware distributed KV manager guides cache eviction and transfer, using signals that come from agent context management in place of the coarse signals earlier systems relied on.
Tests cover coding, math and computer-use tasks, with the models GLM five point two and Kimi K two point six, on up to two hundred and fifty-six GB three hundred GPUs. The comparison is against Dynamo, NVIDIA's production framework. Evolving Infra reduces mean end-to-end request latency by up to four point one times. On P ninety-five time to first token, the reduction reaches fourteen times.
The design separates two things that had been bound together. Once compute allocation and cache management are no longer tied, the balance of prefill and decode can track the workload, and what stays in cache can track what agents will need next.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Agents make strange serving traffic: shifting, bursty and full of context they come back for.
You run a cluster serving language models to agents. Some agents read huge prompts, others write long answers, and the mix shifts through the day. Your nodes are split into a fixed set for prefill and a fixed set for decode.
The road ahead
- 1 · The problem ~7 min Agent workloads shift between prompt-heavy and output-heavy, yet clusters fix which nodes read prompts and which write answers, and evict cache by recency.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Free each node's cache from its role, size the prefill and decode split with online models, and evict cache no active agent branch references.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Decouple cache from role, switch nodes in place, size the split with online models, then evict and route by agent branch counts.
- 6 · The experiment ~6 min You’ll guess: Against Dynamo, NVIDIA's production framework, by up to how many times did Evolving Infra cut P95 time to first token?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.