TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
Shuhuai Li, Shengdong Ye, Dongdong Ge, Yinyu Ye, Jianghao Lin
Shanghai Jiao Tong University · Stanford University
Published October 27, 2026 · on the feed October 29, 2026 · ▲ 95 · arXiv 2610.37230
The abstract
Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems (1) overlook tool-return readiness in service evaluation, wasting limited resources, and (2) use heuristic strategies to decide admission and placement on GPUs, resulting in suboptimal scheduling decisions. To bridge this gap, we present TAO, a tool-aware joint scheduling framework for LLM-based agentic serving. Its value interface combines near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint; a dynamic program jointly selects and places waiting sessions under capacity constraints. Across the 12 reported model-workload-concurrency settings, TAO reduces latency by up to 26.8%, reduces mean time to first token (TTFT) by up to 36.3%, and increases decode throughput by up to 29.5% relative to the best evaluated state-of-the-art baseline for each metric.
The Brief
1:33A short podcast summary of the paper. Written and voiced by AI.
Transcript
TAO. Tool-Aware Joint Scheduling for LLM-Based Agentic Serving. By Shuhuai Li and four others, from Shanghai Jiao Tong University and Stanford University. Published on the twenty-seventh of October.
An agentic service keeps switching between two kinds of work: model inference on clustered GPUs, and external tools running on distributed CPUs. Existing serving systems handle this badly in two ways. They overlook whether a tool has returned when they evaluate a service, which wastes limited resources. And they rely on heuristics to decide which sessions to admit and where to place them on GPUs, which leads to suboptimal scheduling.
TAO is a tool-aware joint scheduling framework for this setting. Joint is the key word: admission and placement are decided together, by one optimisation, rather than by separate rules of thumb.
Each waiting session is scored through a value interface that combines three things. Near-term tool readiness. A node-specific cache credit that decays over time. And the session's token footprint. A dynamic program then selects which waiting sessions to run and places them, all under capacity constraints.
Results cover twelve reported settings of model, workload and concurrency. Against the best evaluated state-of-the-art baseline for each metric, TAO cuts latency by up to twenty-six point eight percent. Mean time to first token falls by up to thirty-six point three percent. Decode throughput rises by up to twenty-nine point five percent.
A scheduler for agents has to know about the tools, not only the model. By folding tool readiness, cache value and memory cost into one score and choosing jointly, TAO turns waiting on tools into a scheduling signal.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Agents keep stepping away from the GPU to run tools. The scheduler doesn't notice when they'll be back.
You run a service where coding agents alternate between model calls on a GPU cluster and tools, like compilers, on separate CPUs. While a tool runs, the session needs no GPU.
The road ahead
- 1 · The problem ~6 min Agent sessions keep leaving the GPU to run tools, yet the scheduler hands out GPUs without asking when each tool will finish.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Give each session a value on each node from tool readiness, decaying cache credit and token footprint, then choose admissions and placements together.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Score each waiting session's readiness, cache credit and footprint, combine them into one value, then let a dynamic program select and place.
- 6 · The experiment ~6 min You’ll guess: Across the 12 main settings, by up to how much did TAO cut mean time to first token against the best evaluated baseline?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.