Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models. By Guoming Ling and seven others, from Sun Yat-sen University. Published on the thirteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Benchmarks are how large language models get evaluated and improved. But many widely used benchmarks are redundant, and that redundancy makes evaluation more expensive than it needs to be. Benchmark compression is the response: evaluate only part of a benchmark, and estimate the full-benchmark scores and rankings from that partial result. The area has grown quickly since two thousand twenty-four, and to the authors' knowledge no survey had yet organized its methods and findings systematically. This one reviews one hundred and nineteen papers.

The review is built around three questions: how methods choose what to evaluate, how they recover full-benchmark results, and how they verify that those results are reliable. From that analysis comes a taxonomy. It covers compression within an individual benchmark, compression across suites of benchmarks, and compression during adaptive evaluation, alongside work on redundancy and on reliability. The methods are then compared on what information they need, their total costs, the reliability checks they use, and their limitations. That comparison is where the survey becomes usable: two methods in the same branch can differ sharply in what they require before they can be run at all.

Three strategies recur. Methods cut cost by selecting representative subsets, by using information that is cheaper or can be reused, and by testing adaptively. Each saving carries a matching risk: poor coverage, inaccurate estimates, and delays introduced by testing in sequence. A subset can miss what it was meant to represent. Cheaper information can predict the full score badly. And adaptive testing, because it proceeds in sequence, can be slow to finish even when it evaluates less. Every route to a cheaper evaluation trades away some part of what made the full one trustworthy.

The taxonomy comes with a practical guide, aimed at helping researchers choose, validate and deploy these methods for evaluation that is both more efficient and reliable. The title's question has a qualified answer. Often, yes, but testing less is only safe when you can show what was lost.