Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles

Measuring the Checker. Mutation Analysis for GPU-Kernel Benchmark Oracles. By Mingzhe Du and three others, from the National University of Singapore, Nanyang Technological University, and one other institution. Published on the second of September. No code repository is linked to it.

When a language model writes a GPU kernel, a benchmark has to decide whether the kernel is correct. Today that decision is made with a few random inputs and a loose floating-point tolerance, and the verdicts now feed leaderboards and reinforcement-learning rewards. Recent work agrees these checkers are weak, and patches them by hand: extra input distributions, fuzzing recipes, tighter tolerances. What nobody has had is a way to measure whether any of those patches is enough. The checker itself has never been checked.

This paper brings mutation analysis to that question, as an adequacy metric for the oracles in kernel benchmarks. The principle is simple. Deliberately break correct code in known ways, then see what fraction of the breakages a test protocol catches. A protocol that lets faults through is scored down for every one it misses.

Deterministic rules inject many thousands of compilable faults into verified CUDA implementations of one hundred and eighty-eight KernelBench problems. More than seven in ten of those faults come with an independent kill witness, meaning there is separate evidence the fault really changes behavior. Any test protocol can then be scored by the fraction of faults it detects.

The official check misses one in six of the witnessed faults, sixteen point nine percent, and it misses them deterministically. The misses are uneven by family. Only eight point seven percent of arithmetic faults escape, but seventy-eight point six percent of precision faults do.

The metric also explains why. There is a tolerance blind band that grows with the size of a reduction, and there is a measured ceiling on how aggressive the inputs can be, set by legitimate floating-point variance.

It can audit patches too. The strongest existing one, KernelBench-Verified, turns out to gain four points from hidden inputs and four and a half points from tighter tolerance, a split its own authors could not compute. A published fuzzing recipe is exposed as rejecting correct kernels, one hundred and seven times.

On the constructive side, optimizing test suites over the kill matrix reaches ninety-eight percent detection with two inputs per problem, and ninety-four point eight percent on held-out faults. The fault taxonomy the measurement produces teaches a test generator more than the raw faults do.

Across forty-eight whole architectures, the blindness grows with scale, and concentrates in deep, homogeneous pipelines. And two problems prove unrefereeable: their official reference implementations violate the benchmark's own tolerance when compared against fp64. Everything is released as a dataset named KernelBench-M.

The shift is from arguing about checkers to measuring them. Before, each fix was a hand-tuned guess with no score attached. With a fault set and a kill matrix, a protocol gets a number, a patch gets its gain decomposed, and a benchmark problem can even be found to have no trustworthy answer key.