JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making

JockeyBench. Evaluating LLM Agents on Horse-Racing Decision-Making. By Zhongwei Xie and ten others, from the Hong Kong University of Science and Technology, Tsy Capital Limited, and two other institutions. Published on the fourteenth of October. Its code repository is worked out: about two hundred source files, with a readme and a licence.

Some forecasting is not a single guess but a series of them. In repeated event forecasting, an agent has to extract predictive signals from noisy evidence, then evaluate what those signals are actually worth over decision after decision. Getting a forecast right, adapting over time, spending on evidence, and turning a forecast into a good decision are different abilities, and a single score would blur them together.

JockeyBench is a benchmark and evaluation protocol that studies this through historical Japanese horse racing. Evaluation keeps three things apart: forecast quality, a net utility specific to the benchmark, and retrospective betting returns and risk under fixed decision rules.

The protocol has four parts. Evidence is time-gated, so an agent sees only what was available before a race. Tools are priced, so research has a cost. Paired frozen-program controls give a fixed point of comparison. And there are two workflows. In one, the agent writes a predictor program that is then executed. In the other, the agent acquires evidence and makes a fresh forecast for each race.

The study covers twenty-eight model and scaffold configurations on three hundred and twenty-four races. After correcting for multiple testing, paired tests identify seven gains from revision and two declines. Yet every one of the configurations has negative mean profit under capped-Kelly settlement.

A four-model comparison puts the two workflows side by side. The reusable program workflow earns higher mean betting profit and pays lower research fees, although both workflows lose on average. Further diagnostics show that mean probability bias reverses on the bets actually selected, and that individual runs which did turn a profit can depend on a few concentrated returns.

Revision produced measurable gains in some places, and still not one configuration made money on average. That gap between improving and profiting is why the authors argue that predictive competence, adaptation, resource use and decision value each need their own measurement when agents are judged on repeated forecasting.