UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents

UserArena. Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents. By Jujia Zhao and eight others, from Leiden University, Renmin University of China, and two other institutions. Published on the twenty-second of October. Its code repository is public and real: about twenty-five source files.

An agentic recommender pursues a shopper's goal through a sequence of decisions, each depending on what came before. To build and evaluate such a policy you need a faithful recommendation world model: one that captures the interactive shopping environment and also how users respond over a long interaction. The resources available rarely offer long-horizon, fine-grained trajectories from real users. They rarely support evaluation at the level of whole trajectories, and they rarely let you validate offline against what a production A/B test actually found.

UserArena is a real-world e-commerce benchmark built to fill that gap. Its target is the user simulator: how faithfully a model can stand in for real shoppers across a long session. Evaluation runs at five levels, climbing from whether outputs are valid, through local decisions and sequential behavior, to personalization, and finally to outcomes.

The data are sixty-four thousand seven hundred and twenty-nine session trajectories, from eleven thousand and ten users. They span five hundred and sixty-six thousand one hundred and two items, and thirteen action types.

Those trajectories are paired with an interactive environment, so the data are not only a record but somewhere a simulator can be run. The benchmark also supports offline validation against the outcomes of a controlled production A/B test, which gives the top level of evaluation a real result to be checked against.

Alongside the benchmark, the authors propose a way to make simulators more faithful over long interactions, called On-Policy Trajectory Distillation, or OPTD. It uses trajectory-level intent to give dense teacher targets on rollouts the student generates itself. Under UserArena's evaluation protocol, OPTD reaches the strongest overall fidelity among the simulators evaluated. It also most closely reproduces the production policy effect when that A/B test is replayed in offline simulation.

Without long real-user trajectories, a shopping simulator could not be checked over whole sessions or against a live experiment. UserArena supplies both: the sessions of eleven thousand real users, and a production A/B outcome to set the simulator's verdict against.