ReproBridge: Building Reproducibility into ML Research Artifacts

ReproBridge. Building Reproducibility into ML Research Artifacts. By Yubo Wang and four others, from the University of Waterloo and Google. Published on the fifth of November.

Reproducing a machine learning experiment means reconstructing a procedure from pieces scattered across a paper, its released code, and whatever supporting artifacts came with it. When a coding agent does the reproducing, that scatter has a consequence. Whether the agent succeeds can depend on how well it fills in the details nobody wrote down, rather than on the released artifacts themselves. Two agents given the same materials can land in very different places, simply because one guessed the missing steps better. And whatever one agent recovers along the way is lost when the session ends.

The question the authors pose is whether keeping those recovered procedures and decisions makes the next, independent reproduction go better. To make the question concrete they define a standardized artifact design with four parts: a refined codebase, an environment setup, experiment launchers, and a reproduction guide. ReproBridge is an agent workflow whose job is to construct artifacts of that design, turning one round of reconstruction into something a later agent can pick up directly.

Testing it needs a benchmark, and ReproBridge-Bench supplies one. It holds one thousand and twenty-nine tasks, drawn from three hundred and eighty-seven experiments across thirty-six papers. Each task is anchored by five human-verified reference scores. Agents attempt the tasks in fresh sessions, given either the raw code or the refined artifacts, and a separate Auditor checks the execution evidence, so a claimed success has to be backed by what actually ran.

Four models were evaluated. With refined artifacts, mean reproduction success rises from seventy-six point four percent to ninety-two point six percent, and downstream API spending falls by nineteen point seven percent on average.

The gap between models shrinks as well. The spread in success rate across models narrows from twenty-one point six percentage points to three point nine, and every one of the four models clears a mean success rate of ninety percent. An ablation removes each component of the artifact in turn. Removing any of them lowers success, and the largest losses come from taking away the code and the environment.

Read together, the results point at execution knowledge as the thing to keep. A model that struggles with raw code is not left behind once the procedure has been preserved for it. It reproduces experiments more reliably, and it spends less on API calls getting there. The artifact carries forward what the first reconstruction learned, so the next agent starts from the recovered procedure instead of guessing at it again.