Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?

Schrödinger's Code Repository. Have LLMs Learned SWE-bench or Memorized It?. By Silin Chen and five others, from Shanghai Jiao Tong University, Xi'an Jiaotong University, and two other institutions. Published on the twenty-first of August. Its code repository is worked out: about one hundred and sixty source files, with a readme and a licence.

Repository-level coding benchmarks have become the standard for evaluating coding agents. They carry an inherent risk of data leakage, because they are built on popular open-source repositories that are used repeatedly for training. Strong performance may therefore reflect memorization of canonical repository cues rather than robust reasoning about the repository. An agent that has seen a repository many times in training may know where a bug lives from the file names alone. A high score would then say more about familiarity than about skill.

SchrodingerRepo, short for Schrödinger's Repository, is an evaluation framework that tests coding agents under repository representations instantiated dynamically. It does not reuse one static representation of the test repository. It treats the repository as a latent variable at evaluation time, instantiated only when the agent enters the evaluation environment. Like the cat in the famous thought experiment, the repository has no settled form until it is observed. There is no fixed copy for a model to have seen in advance, because each evaluation produces its own.

The instantiated repository preserves the original executable behavior, while familiar cues such as naming conventions, file layouts and implementation patterns are eroded. There are four transformation levels: reconstructing the problem statement, remapping the namespace, reordering the layout within files, and rewriting code while preserving functionality. Popular language models are evaluated on SWE-bench Verified and SWE-QA. The levels move from the surface toward the substance. The first changes only how the problem is described, the middle two change names and arrangement, and the last changes the code itself while keeping what it does.

Removing familiar repository cues consistently degrades agent performance, and substantially increases interaction costs, across models. Further analysis traces the added cost mainly to increased difficulty in exploring the repository and localizing within it. Put simply, when the repository no longer looks the way it usually does, agents take longer to find their way around it and to find where the problem is. Since the executable behavior is unchanged, the task itself is no harder. Only the familiarity is gone.

Current coding agents, the authors suggest, may partly rely on memorized, repository-side cues. If so, a benchmark whose repository looks the same every time it is run measures some recall along with reasoning, and the authors argue for evaluation under repositories instantiated fresh at test time.