BizBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business
Enhan Zhao, Yongqi Tong, Xuejun Yu, Hang Wang, Pan Wang, Yichao Wang, Di He, Jianshe Li and 1 others
Peking University · Ant International
Published October 14, 2026 · on the feed October 19, 2026 · ▲ 492 · arXiv 2610.20424
The abstract
Existing long-horizon agent benchmarks often emphasize final task success while providing limited assessment of intermediate execution validity. Yet real-world business operations require agents to navigate extensive, potentially outdated or conflicting policies while satisfying critical constraints throughout execution. These requirements are particularly consequential in high-stakes domains such as payments, where intermediate violations can incur financial losses or regulatory consequences regardless of the final outcome. To bridge this evaluation gap, we introduce BizBench, a multi-turn benchmark grounded in real-world procedures from a leading payments company. Its executable environment integrates 789 knowledge documents and 601 discoverable business tools, with consistent policies, states, and tool behavior. The benchmark comprises 50 tasks involving resource constraints, temporal dependencies, prior commitments, and conflicting or outdated guidance. Each task is validated through reference execution, while deterministic checks assess both goal attainment and execution validity without prescribing a single solution path. Experiments reveal substantial limitations even among frontier models: Claude Opus 5 and GPT-6 Sol achieve strict success rates of only 64.0% and 61.3%, respectively. For Qwen3.8-27B, goal attainment reaches 42.0%, but strict success drops to 32.7%, demonstrating how outcome-based evaluation can mask invalid execution. Beyond success rates, DeepSeek-V4.1-Flash averages 191.3 tool calls per run, approximately 70% more than reference executions, revealing substantial execution overhead. Overall, BizBench establishes a testbed for evaluating and advancing long-horizon agents beyond task completion, toward reliable and constraint-compliant execution in complex business environments.
The Brief
2:46A short podcast summary of the paper. Written and voiced by AI.
Transcript
BizBench. Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business. By Enhan Zhao and eight others, from Peking University and Ant International. Published on the fourteenth of October. Its code repository is worked out: about four hundred source files, with a readme and a licence.
Long-horizon agent benchmarks tend to ask one question: did the agent finish the task? They say much less about whether the steps along the way were valid. Real business operations care about those steps. An agent has to work through policies that are extensive, possibly outdated, sometimes in conflict with each other, and it has to respect critical constraints the whole way through. In a domain like payments, that matters more than usual. A violation in the middle of a run can cause financial loss or regulatory trouble, whatever the final outcome turns out to be. A benchmark that grades only the ending cannot see any of that.
BizBench is built to grade both. Deterministic checks assess goal attainment, meaning whether the agent got where it needed to go, and execution validity, meaning whether it got there without breaking anything on the way. Those checks do not prescribe one solution path, so an agent is free to take a different route from the reference, as long as the route is valid. Strict success requires both at once.
The benchmark is multi-turn, and it is grounded in real procedures from a leading payments company. Its executable environment holds seven hundred and eighty-nine knowledge documents and six hundred and one discoverable business tools, with policies, states and tool behaviour kept consistent across them. On top of that sit fifty tasks. They involve resource constraints, temporal dependencies, prior commitments, and guidance that conflicts or has gone out of date. Every task is validated through a reference execution before it is used.
Frontier models leave a lot of room. Claude Opus 5 reaches a strict success rate of sixty-four percent. GPT-6 Sol reaches sixty-one point three percent.
The gap between the two kinds of grading shows up most clearly with Qwen3.8-27B. Its goal attainment is forty-two percent, but its strict success falls to thirty-two point seven percent. The difference is runs that reached the goal by an invalid route, which an outcome-only benchmark would have counted as wins. That is the masking the benchmark set out to expose. Efficiency is measured too. DeepSeek-V4.1-Flash averages one hundred ninety-one point three tool calls per run, about seventy percent more than the reference executions need. That is a substantial execution overhead, on top of the validity problem.
Finishing a task and doing it properly are separate properties, and the Qwen result puts a number on the distance between them. BizBench is offered as a testbed for pushing long-horizon agents past task completion, toward execution that is reliable and stays inside its constraints, in business settings where the path counts as much as the destination.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
In payments, how you get there matters as much as getting there.
You are evaluating agents that will handle customer requests for a payments company. Long-horizon agent benchmarks often ask mainly one thing: did the agent finish the task?
The road ahead
- 1 · The problem ~6 min Grading an agent only on whether it finished a payments task misses rule breaks that happen along the way and still cost money.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min BizBench grades both whether an agent reached the goal and whether every step stayed within the business rules, using programmed checks.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min BizBench turns real payments procedures into an environment, builds linked tasks on it, proves each task doable, then scores runs on outcome and process.
- 6 · The experiment ~6 min You’ll guess: On BizBench, what strict success rate did the best frontier model tested, Claude Opus 5, reach?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.