AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations

AutoVend. Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations. By Qiming Shi and nine others, from StepFun, Zhejiang University, and four other institutions. Published on the twenty-seventh of October.

Agents built on large language models are moving from shopping on behalf of consumers to running businesses on behalf of sellers. Online-to-offline commerce makes that shift costly. Decisions are issued through Web-based platforms, but they are carried out through physical execution that costs money. Judging an agent one transaction at a time misses the point. What needs asking is whether decisions made online turn into sustained economic returns.

AutoVend measures exactly that, over a long horizon. A single agent manages twenty vending machines across five site types, working through a Web-based operating interface, for one hundred and eighty simulated days. The outcome that counts is economic. Alongside raw returns, the benchmark also looks at returns per dollar of inference cost.

Underneath sits a data-grounded simulator built to make operations expensive in realistic ways. There is a shared warehouse. Procurement has lead times. Replenishment is paid for. Inventory expires. Operating costs recur. Five agents are evaluated against a rule-based inventory-control baseline, and the analysis goes past the final score to diagnose site coordination, execution, and performance over time.

Four of the five agents fall short of the rule-based baseline. Measuring returns per dollar of inference cost also reorders the field: models rank differently on that measure than on returns alone. Operational analyses show that gains in individual metrics do not consistently produce stronger economic growth, because inventory and execution costs erode them.

Trajectory analysis then points to where things go wrong. Agents have trouble grounding actions that are actually feasible, trouble managing the inventory commitments they have made, and trouble converting feedback into policy revisions that work.

A rule-based controller beating four of five agents is the headline, but the operational result explains it. Improving one metric is not the same as growing a business when every action carries a physical cost, and a benchmark that runs for a hundred and eighty days is long enough for those costs to show.