ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval

ExecRetrieval. Measuring the Functional-Correctness Gap in Code-Embedding Retrieval. By Aaryan Kapoor and one other, from Kennesaw State University. Published on the first of September. No code repository is linked to it.

Embedding-based code retrieval sits at the core of coding agents and of retrieval-augmented code generation. In that setting, retrieving correct code matters more than retrieving code that is lexically similar. Existing code-retrieval benchmarks, though, do not plant controlled near misses in the search pool: variants of each query's canonical implementation that differ by a single edit, with execution confirming they are wrong. So one question goes unanswered in a retrieval setting. Can embeddings functionally tell correct code apart from a near clone that is incorrect?

Answering it requires a benchmark whose search pool itself holds the relevant counterfactuals: buggy variants, verified by execution, that are nearly identical to each canonical solution. With those in place, a retriever's rank ordering can be tested directly for functional discrimination, rather than for topical overlap or for recognising the same code it has seen. The scores reported are exec at ten and exec at one, which, as their names suggest, look at the top ten results and at the single first result. The gap between those two cutoffs is where the story of the benchmark lives.

ExecRetrieval contains nine hundred and thirty-nine Python tasks. Each is paired with one canonical implementation verified by execution, and with up to four buggy distractors, also verified by execution. Every distractor is generated by a mechanical mutation that makes one targeted edit. So the wrong answers in the pool are as close to the right one as a single change allows, and execution, not appearance, is what separates them.

The evaluation covers twenty-three dense embedding configurations plus BM25, each invoked the way its provider natively invokes it. The statistics use paired McNemar tests and bootstrap intervals at the level of individual queries.

With near-clone counterfactuals in the pool, the top hosted system reaches an exec at ten of one point zero zero. Its exec at one is only zero point three three one. Across the four leading systems, when the first-ranked result is a miss, it is a paired buggy variant between ninety-one point five and ninety-nine point four percent of the time.

Looked at from the canonical side, the correct implementation scores below at least one of its own four paired distractors in sixty-seven to seventy-eight percent of queries on the leading systems.

The distance between those two numbers for the top system is the gap in the title. Within the top ten, the correct code was always found. In first place, a broken twin usually took the spot. A retriever can look nearly perfect by the looser measure and still mostly hand back the wrong version of the function. The full dataset, the execution oracle, the embedding matrices, an environment snapshot and the pairwise statistical tests are all released.