MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code

MetaKernelBench. Measuring GPU Kernel Knowledge Transfer Beyond Code. By Xueyi Chen and seven others, from Westlake University, Shanghai Jiao Tong University, and one other institution. Published on the fourth of October. Its code repository is worked out: about fifty source files, with a readme and a licence.

Agents that optimize GPU kernels have started keeping what they learn. Some store it in knowledge bases, others as distilled skills. Yet the benchmarks these agents are scored on look only at the implementation in front of them: is each attempt correct, and how fast does it run. Whether the experience an agent retained has any value on a later attempt is left unmeasured. A benchmark can tell you an agent wrote a good kernel. It cannot tell you whether the lesson it took away from writing that kernel helps the next time.

MetaKernelBench sets out to measure exactly that reuse value. The question it asks is narrow on purpose. Take experience distilled from an attempt in one kernel domain-specific language, or DSL. Does it improve a fresh attempt at the same problem in a different DSL? Holding the problem fixed and changing only the language means any difference comes from what was carried across, not from a change of task.

The benchmark has seventy-four problems, each a fused subgraph, grouped into six families. Every problem is posed as a pair of variants, one in CuTe DSL and one in TIRx, and the two differ only in the DSL. An agent first attempts each variant on its own, solo, and is told to distill what it learned into a natural-language skill. That skill is passed across regardless of whether the source attempt passed verification. It is then the only extra input given to a skill-conditioned attempt by the same model in the other language. Each skill-conditioned attempt is compared against the solo attempt on the same variant, with matched budgets per attempt, scoring both correctness and end-to-end runtime.

Six models were tested, in both directions. Paired lift over the solo attempts ranges from minus nineteen percent to plus twenty-nine percent. Four of the six models gain in both directions. Yet every model, in every direction, also regresses on some problems: between sixteen percent and forty-five percent of them.

What predicts the outcome is not simply whether the source attempt succeeded. It is how the source attempt's result stands relative to the target's own solo attempt. When the source stands above, the transferred skill improves the target in seventy-one percent of comparisons. When it stands below, the target regresses in fifty-four percent.

So retained experience cuts both ways. A skill carried from a stronger attempt tends to help, and one carried from a weaker attempt tends to drag the target down, which is why the same model can gain on average and still lose on a sizeable share of problems. The authors present MetaKernelBench as a complement to implementation-quality metrics: one score for the kernel that was written, and a separate measure of whether same-problem knowledge survives the move from one DSL to another.