Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents. By Zilin Wang and four others, from the Hong Kong University of Science and Technology, Guangzhou, Yinwang, and two other institutions. Published on the fifteenth of October. Its code repository is worked out: about seventy-five source files, with a readme and a licence.

Large language model agents solve tasks through long sequences of tool calls, and the prevailing design has a single strong model generate every step, even though many steps are routine enough for a much smaller model. Existing ways of sharing this work keep the strong model as the generator. Routers and collaborative agents decide which model handles each query or each step. Speculative methods check a small model's drafts one step at a time, usually against the action the strong model itself would have taken. Either way, the strong model is still the one producing the trajectory, step after step, and it pays the cost of decoding each of them. The smaller model may help, but the expensive work of writing the trajectory has not moved.

SpecAgent changes the strong model's job. It is a collaboration protocol that turns the strong model from the author of each step into a reviewer of steps drafted by a weak model. Two shifts follow from that. Speculation now covers a stretch of trajectory rather than a single step, and the check asks whether a step is semantically valid, rather than comparing it with an action the strong model would have written itself.

The weak model executes read-only tool calls as it drafts, so every step it proposes is grounded in real observations rather than guesses about what a tool would return. It holds back the first action that would modify the environment. The strong model then verifies the drafted segment in a single prefill pass. It commits the longest prefix it judges semantically valid, and repairs the first step it rejects.

The saving comes from that verification step. Because verification replaces autoregressive decoding with parallel prefill, the strong model decodes only short verdicts and occasional repairs, rather than every step of the task. Reading many drafted steps at once is cheaper than writing them one token at a time.

On tau-squared bench, SpecAgent comes within one point of the strong model's success rate, while roughly halving memory traffic and wall time. On AutomationBench and VitaBench, it recovers between forty-four percent and seventy-five percent of the quality gap, while lowering both costs. The gap here is the distance between what the weak model reaches alone and what the strong model reaches alone, so recovering part of it means landing in between, closer to the strong end, at lower cost.

The authors also test an edge-cloud deployment on MCP-Atlas with three cloud models. There it closes between fifty-three percent and seventy-five percent of the quality gap between the weak and strong models, while the strong model generates at most half as many tokens. In that setting the weak model drafts at the edge and the strong models review from the cloud, which is the same division of labour carried across a network.

The conclusion drawn is that a strong model need not author every step of an agent. Reviewing drafts that are already grounded in real observations retains most of its quality, while cutting both latency and memory traffic. The boundary of each draft sits at the point of no return. Drafting runs freely through actions that only read, and stops before anything that would change the environment, which is exactly where a second opinion matters most.