Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Benchmark Radar. A Living Database and Search Engine for AI Benchmarks and Evaluation. By Koutian Wu and six others, from Earth Space AI, Tacite AI, and eight other institutions. Published on the tenth of September. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.

Benchmark Radar is a living database and search engine for AI benchmarks and evaluation. It is built for benchmark researchers and for developers of large language models and other AI systems, who share three needs: finding relevant evaluations, locating the datasets and code behind them, and understanding the settings behind a reported score. A number on a leaderboard means little without knowing which benchmark produced it, where that benchmark lives, and under what conditions it was run. Benchmark Radar is meant to put those answers in one searchable place, and to keep them current.

Coverage spans evaluation of language models, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system joins several parts. Daily discovery of benchmark papers, repositories, datasets and releases. A searchable benchmark catalog. Mentions of benchmarks in model cards and technical reports. And histories of scores. Source identities and citations are kept throughout, so a reader can inspect a candidate benchmark alongside its evaluation evidence.

What is released is a web dashboard with a benchmark leaderboard, a Pareto frontier view plotting score against measured use, views of saturation and trends, and daily feeds. It also offers downloadable evidence, a command-line interface for offline queries, and reproducible analysis.

Daily discovery draws on thirty-seven sources: thirteen direct connectors, and twenty-four first-party research and engineering feeds. The catalog contains one thousand two hundred and eighty-three source records, drawn from four benchmark catalogs. Of those records, seven hundred and ninety carry numeric observations, twelve thousand nine hundred and sixteen of them in all. The authors describe the collection and retrieval process and audit the full catalog.

Anyone designing a new evaluation is the clearest audience. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence before building something new. The catalog is also put to use directly, to examine benchmark saturation, adoption trends, and the limits of comparing scores. Those three questions matter to anyone reading results as well as building evaluations: whether a benchmark still separates models, whether anyone is actually using it, and whether two reported numbers can fairly be set side by side. Model developers, who need to find the right evaluations for their systems, are the other intended users.

Two properties separate this from a static list. It keeps moving, with new material discovered daily, and it keeps its sources attached, so a score can be traced back to where it came from rather than taken on trust.