τ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction

τ^τ-Bench. An Environment for End-To-End, Realistic Agent Construction. By Quan Shi and three others, from Sierra and Princeton University. Published on the fourth of September. Its code repository is worked out: about five hundred and fifty source files, with a readme and a licence.

Language-model agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes and operate internal systems. The work of building them is increasingly handed to coding agents. What existing benchmarks say little about is whether an AI system can deliver one under the conditions of a real client engagement, which is a different question from whether it can write code that runs.

Hyper-tau-bench makes agent construction itself the task. A developer agent is given the things a real engagement gives you: the records a business actually keeps, a client who holds the requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and on models. That list is the design principle stated outright: the same starting point a real engagement provides. From it the developer agent has to deliver a complete customer-service agent.

The scoring is what gives the benchmark its teeth. The delivered agent is not inspected or graded on its code. It is deployed against held-out simulated users, and how that deployment performs is the developer agent's score. Fifty-three tasks span four domains, and four rather than one keeps the result from being a property of a single business. Because the requirements live with a client rather than in a document handed over at the start, getting them out is part of the task.

The strongest configuration tested, Claude Opus 5 running under Claude Code, passes just twenty-three point nine percent of evaluation simulations. An expert-authored reference ceiling scores eighty-two point two percent, so the gap is not the benchmark being impossible. The failures mirror ones human agent developers see: shallow queries issued in place of deep comprehension of the records, almost nothing communicated to the client, and too little experimentation with agent architecture and serving spend, shipping the first design that runs.

Those three failure modes have something in common. Each is a failure of engagement rather than of coding: not reading enough, not asking enough, not trying enough alternatives. That is what the authors are aiming at, turning the work of cooperative agent building into a measurable target for coding agents, which is where this work is already being sent.

τ^τ-Bench. An Environment for End-To-End, Realistic Agent Construction. By Quan Shi and three others, from Sierra and Princeton University. Published on the fourth of September. This is the long version: the whole paper rather than the abstract.

Picture what the developer agent is handed. A customer service handbook says a card replacement fee applies to standard accounts, with tier exceptions that refer you to a servicing matrix and warn you not to quote the public site. The public fees page lists a lower number and was last updated fourteen months ago. A support transcript shows an agent waiving the fee. An email from the operations team says legal confirmed the higher figure, waived only for the premier tier. A chat channel has one person noting the public site still shows the old number and another replying that it is stale. And a recording says the same thing out loud. Four sources, three answers, and the truth is recoverable but scattered.

That is the first of two axes of difficulty the paper identifies. Recovering the specification is hard because the knowledge arrives as a business actually keeps it, across procedures, transcripts, spreadsheets and email threads, and because much of it lives only in human stakeholders whose requirements surface under questioning. The second axis is construction. A developer rarely starts from a blank slate or an unbounded budget: there may be an inherited codebase whose behaviour must be preserved, a client-supplied interface that may be subtly defective, hard limits on cost and latency, and a fixed menu of models to serve from.

The environments are built by decomposing each domain's operating procedures into atomic facts and then scattering those facts across the artifacts a business would actually hold: handbooks, support transcripts, images, audio, fee schedules in spreadsheets, email threads. What remains after that scattering lives with a simulated client the developer can interrogate. The developer works in a sandbox holding only those artifacts, and submits a complete agent, which is then run against held-out tasks with a simulated user pursuing a hidden goal. The score is whether the final state matches the expected one.

The strongest configuration measured passes twenty-three point nine percent of evaluation simulations, against an expert-authored reference ceiling of eighty-two point two. But the aggregate hides where the failure concentrates. The same developer averages fifty-five point nine percent on airline, seventy-two point eight on retail, and forty-eight point two on telecom, and then five point nine percent on banking. The reason is scale of evidence. Those first three domains span eighty-five, a hundred and nineteen, and a hundred and fifty-five atomic facts. Banking carries two thousand nine hundred and sixty-nine, and a single banking task can draw on up to five hundred and eighty of them, which is more than an entire other domain.

The trajectories explain the collapse better than the totals do. On banking's whole-domain corpus of over seventeen hundred files, one developer reasons that reading it all is impractical and pulls the policy out with targeted searches instead. Later, in evaluation, the agent it built tells a user it has no verified catalogue of the bank's card offerings. And across all four harnesses, after an opening stretch of reading and searching, the builds settle into writing code and running self-tests, with talking to the client accounting for three tenths of one percent of every tool call made.

Some caveats deserve to travel with the number. The client is simulated, so its willingness to answer and the shape of its answers are design choices rather than measurements of real stakeholders. Fifty-three tasks across four domains is not a large sample, and one domain dominates the difficulty so completely that the aggregate score is close to a statement about banking. Build costs vary enormously between harnesses, from around forty-eight minutes to over three and a half hours, which makes cross-harness comparison partly a comparison of how long each was willing to work. And the expert ceiling is itself an artifact, authored by people who knew what the evaluation would ask.

The framing that survives is the one in the analysis section. The gap is traced to the work that makes agent building a research problem rather than a conventional engineering task: gathering requirements, exploring designs, running experiments, validating against ground truth. Today's systems produce agents that run and not agents ready to deploy, and every one of the five named failures is a failure of inquiry rather than of code. They searched the evidence instead of reading it, shipped the first architecture that worked, and did not ask.