CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

CARTBench. Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines. By Yuqin Dai and eleven others, from Zhejiang University, Xiaohongshu, and two other institutions. Published on the fourth of November.

Shopping assistants built on large language models now recommend products over multi-turn conversations. Evaluating them reliably runs into two obstacles, which the authors name directly. One is data scarcity. The other is the difficulty of standardising criteria: what counts as good advice is hard to pin down in a way that holds from one case to the next.

CARTBench is named for CART, which stands for Complex and Ambiguous Requests with Timelines. It asks how well an assistant handles shopping needs of that kind. Its criteria fall into three groups: Core, Excellence and Safety. Core captures the behaviour that is required, and Excellence the step up to genuinely high-quality purchase advice.

Each obstacle gets its own answer. Against data scarcity, cases are curated from real commercial shopping needs across seventeen retail domains, pairing timestamped dialogue with one thousand six hundred and three product records. Against vague criteria, each case gets its own rubric, guided by Ideal Response Principles. Those principles were distilled from excellent replies, chosen by hand from orders that were completed successfully.

The same principles also drive the authors' own recommendation strategy, Principle-Guided CART Prompting. In total the benchmark contains one thousand three hundred and twenty-eight expert-annotated criteria. The models evaluated are seven flagship commercial API models, among them GPT-6.1-Sol and Claude-Opus-5, alongside open-weight models of thirty-five billion parameters and up.

In the first full-suite evaluation under the frozen rubrics, three of the ten models exceed ninety percent on Core. None exceeds forty-five percent on Excellence. The authors call this an understanding-to-execution gap, lying between the behaviour that is required and purchase advice of high quality. Diagnosing the failures, they find shared weaknesses in product evidence and in needs left unresolved, tied to requirements that change over the conversation and to decisive unknowns.

Clearing the required bar and giving excellent advice are measured separately here, and the scores show how far apart they sit. The failures the authors trace point to where a real shopper's request is messiest: requirements that move, and something decisive that is still unknown.