E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

E-Commerce Bench. Evaluating LLM Agents on Long-Horizon Autonomous Business Operation. By Wei Fan and ten others, from the Qwen Team at Alibaba Group, Taobao and Tmall Group at Alibaba, and one other institution. Published on the thirty-first of August. Its code repository is worked out: about fifty source files, with a readme and a licence.

Long-horizon agentic tasks are more than short tasks chained over more turns. Their environments evolve and their dependencies stretch far, so a large language model has to keep exploring, learn from experience, and adapt its policy over thousands of steps. A task that ends after a few dozen moves cannot show whether a model does any of that. Testing for it needs a setting long enough, and changeable enough, to demand it. E-Commerce Bench is presented as the first open-source benchmark that brings multi-round negotiation with counterparts and dynamic events together inside a year-long business operation.

The task is to run a business. Over a three hundred and sixty-five day year, an LLM agent operates several online stores at once. It researches the market, negotiates with suppliers to source inventory, optimizes sales strategies, fulfils orders, handles returns and manages cash flow, all to maximize its total assets at the end of the year. Year-end assets are one of seven dimensions the evaluation reports. Others include fraud avoidance and operational efficiency. So the agent is judged not only on how much it ends with, but on how it got there: whether it was cheated along the way, and how efficiently it ran the stores.

The design pursues realism and reproducibility at once. For realism, product and supplier data are derived from a real e-commerce platform, and a year-long calendar of promotions, natural disasters and supply-chain shocks keeps reshaping demand. For reproducibility, both sides of the market are deterministic. Customer purchases and returns follow a fixed demand model. Supplier pricing, concessions and decisions are set by a negotiation kernel, and an LLM is used only to put those decisions into words. The agent bargains with something that talks like a supplier but decides by fixed rules.

Eighteen frontier models were evaluated, and no single model dominates. Every agent starts the year with the same opening stake of one hundred thousand. GPT-5.6 Sol earns the most, growing that stake to roughly one point four three million by the end of the year. Yet the same model ranks sixteenth of eighteen on fraud avoidance, and it trails Fable5 in operational efficiency. Being the best earner and being the best operator turn out to be different things here.

Among open-weight models, Qwen3.8-Max-Preview leads, finishing with four hundred and sixteen thousand two hundred and fifty-two. That is thirty-eight percent above GLM 5.2 in its high setting. It also shows the strongest learning over the horizon, progressively bargaining prices down across repeated orders.

What stays with you is the split between earning and judgment. The model that made the most money was also among the easiest to defraud. A single year-end number would have hidden that, and reporting seven dimensions is what brings it into view. Learning across the year turns out to be measurable too, in something as plain as the price paid on a repeated order.