IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
Yale University · TCS Research
Published September 9, 2026 · on the feed September 11, 2026 · ▲ 24 · arXiv 2609.10539
The abstract
A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.
The Brief
2:24A short podcast summary of the paper. Written and voiced by AI.
Transcript
IdeaAMBIG. Benchmarking Implementation-Critical Gaps in Research-Idea Specifications. By Yiling Ma and four others, from Yale University and TCS Research. Published on the ninth of September. Its code repository is public and real: about sixty source files.
A research idea can be novel, coherent and scientifically plausible, and still leave its method too loosely specified to build faithfully. The authors study that property, which they call codification readiness. A method specification is ready if it gives a competent implementer, or a coding agent, enough methodological information to construct the intended method without making unsupported assumptions. An idea can pass every other test and fail this one, and nothing usually measures it.
IdeaAMBIG measures three capabilities. The first is codification-readiness assessment: judging whether a specification is ready. The second is defect localization: finding where it falls short. The third is clarification action generation: saying what should be done about the gap. The two later tasks see different inputs. Defect localization receives only the specification. Clarification also receives the annotated defect.
The benchmark holds six hundred and sixty evidence-grounded instances. The specifications, and the resolutions that support them, are built from papers, codebases, issue threads and reproduction artifacts. One hundred and sixty-three of the instances are real-world gaps, drawn from reproducibility reports and GitHub issues. The other four hundred and ninety-seven are controlled synthetic gaps, injected into references that were codification-ready to begin with.
Thirteen LLMs were evaluated. On real-world instances, the best model achieves a Macro Defect Recovery Rate of just nine point six percent. Yet given the defect, the best Macro Clarification Action Success Rate reaches eighty point six percent. The pattern holds across all models: localization is the main bottleneck, and clarification is much stronger once the defect is known.
An oracle study shows how much the gaps matter. Supplying the gold resolution raises the downstream codification-ready rate from fourteen percent to ninety-eight percent.
The benchmark separates two skills that are easy to blur. Models can often say what to do about a gap in a method once someone points to it. What they struggle with is noticing the gap at all, and the oracle study shows that filling those gaps is what turns an idea into something buildable.
Paper Quest · 7 chapters · about 36 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An idea can be brilliant and still impossible to build faithfully. That is the wall here.
Research agents now go from idea to code. You hand one a method description that looks fine: novel, coherent, plausible. But it never says which of two reasonable choices the method uses.
The road ahead
- 1 · The problem ~7 min A method description can look complete yet leave out a choice that decides which method gets built, and existing tests largely miss that.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Test whether models can tell a spec is ready to build, find its one gap, and clarify it, each separately.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: Across 13 LLMs, what Macro Defect Recovery Rate did the best model reach on real-world instances, finding the gap from the specification alone?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.