Δτ-Bench: Pushing Agents Toward a Moving Capability

Δτ-Bench. Pushing Agents Toward a Moving Capability. By Shuofei Qiao and eight others, from Zhejiang University and University College London. Published on the twenty-fifth of October. Its code repository is worked out: about one hundred and fifty source files, with a readme and a licence.

Large tool ecosystems do not sit still. Services evolve independently, and as they do, the ways a task can be accomplished keep changing. Familiar tools disappear or get replaced. More efficient alternatives appear, often among tools with overlapping functionality and noisy documentation. An agent that learned one way of doing things has to notice when that way is gone, or when a better one has arrived. The authors argue existing benchmarks do not examine those demands together.

Δτ-Bench is an executable agent benchmark that models this structural evolution directly. Against what the authors call a moving capability frontier, it asks two questions of an agent. Can it proactively discover efficient tool-use paths? And can it promptly revise the procedural knowledge it has accumulated, once that knowledge goes stale?

The benchmark contains one thousand five hundred and eighty-eight validated multi-turn tasks across forty-two domains. The ecosystem passes through six successive states, each generated by capability-preserving transformations. The tasks stay solvable from state to state, but the route to solving them moves.

Even the strongest evaluated model or harness reaches only forty-two point three percent task success. Existing skill-learning methods do produce some initial gains, but those gains do not consistently carry over into better adaptation as the ecosystem keeps evolving.

Those results motivate a proposal of the authors' own, Steward, an ecosystem manager. It jointly manages tools and procedural knowledge, through incremental skill maintenance and dynamic provisioning while a task is running. In their experiments Steward sustains higher task success across ecosystem changes, at lower skill-maintenance overhead. It also improves both recovery from disrupted workflows and adoption of more efficient alternatives.

The benchmark reframes tool use as a moving target rather than a fixed skill. Learning a procedure once is not enough when the tools underneath it keep changing, and the evidence here is that keeping that knowledge current is a separate problem, one that Steward treats as ongoing maintenance.

Δτ-Bench. Pushing Agents Toward a Moving Capability. By Shuofei Qiao and eight others, from Zhejiang University and University College London. Published on the twenty-fifth of October. This is the long version: the whole paper rather than the abstract.

Picture a large company that puts one agent in front of all its services. Behind that single entry point, each department maintains its own tools and updates them on its own schedule. Tools get retired, split or merged, and descriptions go stale. A learned workflow can simply break. The quieter failure is the one the paper cares most about: a newer, cheaper path appears, and the agent never notices, because its old procedure still succeeds. Task success, in other words, can hide a growing gap between what the ecosystem offers and what the agent has learned to do with it.

Existing benchmarks test retrieval, stateful execution, replanning or robustness to tool changes largely one at a time. Delta Tau Bench puts them together. It has one thousand five hundred and eighty-eight validated tasks across forty-two domains, each run against six successive versions of the tool ecosystem, for nine thousand five hundred and twenty-eight task and ecosystem pairs. A simulated user holds a private handbook and reveals details only when asked. Success is judged on the database, not the dialogue: a task passes when the agent's changes cover every change the validated gold execution made.

The key move is that every version change is capability-preserving. A tool may be renamed or given new parameters, split into several coordinated calls, or several consecutive steps merged into one interface. Sometimes the old tools stay alongside the new one, forcing a choice among overlapping options; sometimes they are retired. A version is accepted only if every base capability is still reachable. So the task never becomes impossible. Only the best route to it moves.

Finding that optimum is a covering problem. Each task has an ordered sequence of base functions from its gold trajectory. At a given version, the available realizations each cover some contiguous block of that sequence: a merged tool might cover three steps at once. Dynamic programming partitions the sequence into blocks and picks one active realization per block, minimising either the number of calls or the total cost. Cost here is a score from one to ten per call, assigned by a language model from each tool's definition and implementation. Both reference paths are replayed to confirm they really work.

From those references come the metrics. Success-weighted path efficiency gives a successful task credit for how close its path came to the reference, capped at full credit. Two more metrics look only at tasks whose best path changed between adjacent versions. The improvement discovery rate covers tasks whose optimum got shorter or cheaper: did the agent find the new route? The degradation adaptation rate covers tasks whose optimum got longer: did the agent still cope?

The paper's own system, Steward, follows from the diagnosis. It is a manager that holds tools and skills together. Skills come in three levels: planning skills for task-level workflows, atomic skills for reusable operations, and tool skills for how to call a single tool. Each atomic skill records the tools it depends on. When a version changes, Steward edits the library locally using newly verified trajectories, rather than rebuilding it, and marks a skill inactive when its tools disappear. Inactive skills are kept, not deleted, so they can come back if their tools return. During a task, Steward builds an initial view from a provisional plan, then lets the agent ask for more tools or delegate a search, in which the manager must inspect each tool's schema before recommending it.

Current models fall well short. The best harness, Codex with GPT-5.5, reaches forty-two point three percent success, and no model exceeds thirty percent on either efficiency measure. Retrieval is lopsided: that harness finds ninety point three six percent of the relevant tools, with precision under twenty percent.

The most revealing result is a ladder of controlled conditions. Narrowing the tool set to the domain, then to the task's tools, then to only the relevant current-version tools, raises success and efficiency at each rung. Handing the model the reference path itself on top of that adds almost nothing. The bottleneck, then, is not planning. It is telling valid, efficient tools apart from retired or overlapping ones, and using them correctly.

Existing skill-learning methods help at the first version, then lose their grip. At the third version, EvoSkill and AutoSkill drop to zero on degradation adaptation, while using no skills at all keeps about eleven percent.

Steward has the highest success at every version, averaging thirty-nine point zero four percent against thirty-six point three six for SkillOpt, while the baselines spend up to five point five times as many maintenance tokens. The maintenance ablation is the cleanest test of the thesis. Incremental updating beats full reconstruction on every metric, with thirty-nine point five percent fewer maintenance tokens on DeepSeek-V3.2. On GPT-5.5, full reconstruction even scores below a library frozen at the first version.

The paper has no limitations section, so these points come from what it reports. The environments, tasks, transformations and cost scores are all generated or assigned by language models, and the reference path is optimal only within the constructed realization space, not over every conceivable route. The simulated user is always DeepSeek-V3.2, and the skill-method comparison, including Steward, runs on that one task model. No error bars or repeated runs are reported, so a gap of under three points between Steward and SkillOpt on success should be read with that in mind. The gains on discovery and adaptation are a few points each, real but modest.

The idea to carry away is that for an agent in a living ecosystem, succeeding is not the same as keeping up. A benchmark has to know the current best route to see the difference, and an agent needs its knowledge tied to the tools that knowledge depends on, so that when a tool changes, the right skill changes with it, and nothing useful is thrown away.