Ask, Don't Equip: Question-Driven Coding Agents for Deep Kernel Optimization

Ask, Don't Equip. Question-Driven Coding Agents for Deep Kernel Optimization. By Gabriele Oliaro and five others, from Snowflake AI Research and Carnegie Mellon University. Published on the second of November.

Language model agents for GPU kernel optimization tend to sit at one of two ends of a tradeoff between structure and freedom. Scripted agents fix what the model does at each step, whether through a learned policy, an optimization menu, or fixed roles. They are sample-efficient where their recipe applies, and they stall where it does not. Open-ended agents let a self-directed model mutate a population of kernels, and they pay for that freedom in trials: thousands of evaluations, repeated edits, wrong candidates. Faced with a hard target, both kinds respond the same way, by equipping the agent with more. Parsed profiler output, disassembly analysis, retrieval over documentation and libraries, stores of lessons.

The argument is that both ends lack the same thing, which is direction: a hypothesis about where the remaining gap is. A short question, asked at the right moment, can supply that direction without narrowing what the agent may do next. That is the distinction the paper leans on. A script narrows the agent by deciding its steps. A pile of tools widens what it can see, but does not say where to look. A question points somewhere and leaves the agent free to choose how to get there. KernelSeek is built on that premise. Its task is to make an existing CUDA kernel faster, with no operator-specific knowledge and no faster implementation to imitate.

It has three parts. A general-purpose coding agent. A minimal harness that enforces a verifiable goal. And questions at decision points, which come first from a human, and later from a bank of questions the agent applies to itself. The harness is deliberately small, and its role is to make the goal verifiable, so that every claimed speedup can be checked. The title puts the contrast plainly: ask, rather than equip.

The first comparison holds everything else fixed: the same task, the same evaluator, the same hardware, the same week. A tool-rich autonomous agent, built in the prevailing style, plateaued at zero point five times FlashAttention-3 after one hundred and thirty-three model-driven steps. KernelSeek reached one point one three times. One stopped well short of the reference kernel; the other passed it.

It then produced dependency-free FlashAttention forward kernels for Hopper and for Blackwell. Each is a single CUDA file, with none of CUTLASS, CuTe, cuBLAS or cuDNN, and each is correct on all one hundred and eighty-two scenarios for its architecture. By geometric mean they run at one point one six times FlashAttention-3, and one point four six times FlashAttention-4.

The traces were examined as well. Across one hundred and sixty-two human messages and eleven point five thousand agent tool calls, every large gain followed a question that did one of two things: it moved the agent to a different layer of the system, or it rejected a limit the agent had claimed. None followed a question that supplied information. Across that whole record, the gains tracked direction rather than data.

That last result is the paper's thesis in miniature. The big steps did not come from telling the agent something it did not know. They came from questions that redirected where it looked, or refused to accept where it said it had to stop.