CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments

CRM-Agent-Bench. Benchmarking Tool-Using Agents in Stateful Enterprise Environments. By Monojit Banerjee and two others, from Salesforce. Published on the twenty-third of October. Its code repository is worked out: about eighty-five source files, with a readme and a licence.

Deploying a language agent in a real application asks for more than the right answer. The agent has to execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue. As the authors put it, the gap between saying and doing is the gap between a satisfied customer and a real loss. A benchmark that only checks the final answer cannot see that gap. An agent can say the right thing to the customer and still leave the records in the wrong state, or reach the right state by breaking a rule along the way. Measuring deployability means looking at what the agent did, not only at what it said.

The authors argue that a benchmark for deployable agents should assess four facets together. The outcome the agent reaches. The trajectory it takes to get there. Its operational viability. And its policy compliance. Customer relationship management, or CRM, is chosen because it exercises all four: mutable records meet operational rules, and actions have consequential effects. The benchmark evaluates the complete agent system rather than the model alone, and alongside success it reports scored coverage, latency and cost.

CRM-Agent-Bench is execution-grounded. It holds five hundred and fifty-five stateful, multi-turn tasks, across nine business domains and thirty-six capability tags. Each task supplies its own executable contract. That contract is drawn from checks over actions, arguments, order, response, target state and protected state, and each task declares which of those checks apply to it. Success is the strict conjunction of the declared checks. An agent that reaches the right state but touches a protected record, or does things in the wrong order, has not succeeded.

Sixteen systems are evaluated on the audited final release. Strict success spans sixty-four point three two percent at the bottom to eighty-three point six percent at the top. The leading systems are closely grouped on that number. Even so, they diverge sharply in cost and in per-domain reliability, differences a single success rate would leave invisible. For anyone choosing among those top systems, the headline score settles little. The cost of running them and how they hold up domain by domain are where they actually part ways.

That last result is the argument for the four facets made concrete. Systems that look alike on outcome come apart once cost and reliability by domain are measured too. The authors offer the benchmark as an auditable framework for measuring contract-grounded business-transaction integrity and adherence to task-declared policies, with a portable structure designed for extension as models improve.