TokenCast: Forecasting Token Consumption During LLM Agent Execution

TokenCast. Forecasting Token Consumption During LLM Agent Execution. By Chaoqian Ouyang and nine others, from Sun Yat-Sen University, Rensselaer Polytechnic Institute, and two other institutions. Published on the twenty-eighth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Run a large language model agent on the same task twice, and the tokens it consumes can differ by more than an order of magnitude. The agent picks its next steps from tool feedback and intermediate results, so the path is not fixed in advance. Meanwhile the context keeps growing, which inflates the input of every call that follows. The total cost of a task is therefore hard to predict before execution starts, and any prediction has to be revised as the run unfolds.

TokenCast learns a composable cost representation for each segment of an execution. A segment's representation records two things: its own consumption, and the context growth it introduces.

That second quantity is what makes composition work. A segment that adds a lot to the context does not just cost what it costs. It raises the price of everything after it. When adjacent segments are combined, the cumulative estimate captures the extra input cost that arises because context from earlier segments is re-read by every later call. As the run proceeds and new evidence is observed, the forecast is refreshed, so the estimate made at the start is not the one the run is stuck with. Doing so requires no additional language model calls, which matters for a tool whose job is to keep costs down: a forecaster that spent tokens to predict tokens would be working against itself. On SWE-bench Verified, the mean cumulative prediction time is thirty-two point eight milliseconds per run.

Evaluation spans four task suites and six agent models, ninety-six combinations in all. Against the strongest comparator, TokenCast reduces mean absolute error by fourteen point five percent on average. A better forecast only matters if something acts on it, so the authors also put it to use. In an offline replay of budget control, TokenCast uses twenty-one point three percent fewer tokens on average than a fixed-budget policy, at matched trace completion.

Cost here is treated as something that compounds. A segment is expensive not only for what it spends but for what it leaves behind in the context, and a forecaster that accounts for that inheritance can keep its estimate current as an agent wanders, cheaply enough to run alongside it.