ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
Jeonghye Kim, Minseon Kim, Young Jin Kim, Matheus Pereira, Marc-Alexandre Côté, Alessandro Sordoni, Xingdi Yuan, Zhengyan Shi
KAIST · Microsoft Research Montréal · Microsoft AI
Published September 16, 2026 · on the feed September 17, 2026 · ▲ 62 · arXiv 2609.18805
The abstract
Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.
The Brief
2:13A short podcast summary of the paper. Written and voiced by AI.
Transcript
ProgramDistill. From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks. By Jeonghye Kim and seven others, from KAIST, Microsoft Research Montreal, and one other institution. Published on the sixteenth of September. No code repository is linked to it.
Coding agents are usually evaluated on tasks where the desired behaviour is spelled out, in an issue or in instructions. Practical web development is often different. An agent may have to work out the behaviour from software that already works, and then implement it inside an application that is incomplete. Existing evaluation does not test that skill.
ProgramDistill evaluates coding agents on features that are discovered by interacting with fully functional reference applications. The agent has a working app to learn from, and an incomplete one to fix. That setting reverses the usual order of information. Instead of reading what a feature should do, the agent has to observe it working, and then rebuild it where it is missing.
Applications are factorized into features at different granularities. Each feature is associated with replayable behaviours that can be executed through its gold patch, which is what makes every task verifiable. A pipeline called mine-craft-patch discovers one thousand nine hundred and seventy-five replay-verified behaviours across twenty-six applications, and constructs four thousand and sixty-three tasks, all without human intervention.
Nine frontier coding agents were evaluated. On cumulative workflows in full-application reconstruction, GPT-6 Astra succeeds forty-nine point two percent of the time and Claude Opus 5 twenty-eight point eight percent.
In partial-application reconstruction, difficulty can be dialled by restoration depth. As depth goes from one to eight, one agent's success falls from a hundred percent to sixty-four percent, and another's from ninety-six percent to thirty-two percent. The deeper the restoration, the more of the application an agent has to rebuild, and the steeper the fall.
ProgramDistill ends up as a benchmark whose difficulty can be controlled, that scales without human labelling, and that can diagnose where coding agents break down. The authors also see it as a natural basis for curriculum-based training in future, since the same depth dial that makes tasks harder could order them from easy to hard.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Sometimes the only specification is software that already works. That is the wall here.
You want to test coding agents the way web work often goes: there is a working app, and an incomplete copy missing some features. No issue text says what to build.
The road ahead
- 1 · The problem ~6 min Real web work often means copying behaviour from a working app into an incomplete one, and few benchmarks test agents on that in a cheap, checkable way.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Record each behaviour of the working app as a replayable trace, hide the code behind it to make a repair task, and chain behaviours for harder tasks.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The pipeline explores a working app, keeps behaviours that replay reliably, masks their code into tasks, and has agents rebuild them against the reference.
- 6 · The experiment ~6 min You’ll guess: In partial-application reconstruction, one frontier agent succeeds 96% of the time at restoration depth 1. How high is its success at depth 8?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.