RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

RealSWE. A Compositional Evaluation of Coding Agents under Realistic User Requests. By Gyuhyeong Kim and four others, from Sungkyunkwan University. Published on the thirty-first of August. Its code repository is a placeholder: a readme, and nothing to run yet.

Coding agents are evaluated on the SWE-bench family, whose tasks are built from curated GitHub issues. Those issues are long, structured and information-rich. Real user requests are not. So there is a distribution gap between what these agents are measured on and what they are actually asked to do, and until you measure the gap you cannot say whether it matters.

The paper measures it rather than asserting it. The authors define a six-category information taxonomy and four dimensions of linguistic style, then apply both to real user prompts from a coding chat product and to problem statements from SWE-bench Verified and Pro. Two numbers land hard. Requests carrying only a problem statement, alone or with limited extra context, are eighty-eight percent of real prompts and seven percent of benchmark problems. And eighty-seven percent of real prompts are casually written, against ninety-four percent of benchmark problems being formal.

RealSWE is built from those observations. It is three hundred and eighty-one multi-variant task families derived from SWE-bench Verified and Pro. The design choice that makes it useful is that variants inside a family share the same underlying task and the same gold patch, differing only in information composition and linguistic style. So a change in score is attributable to how the request was phrased and what it included, and not to the difficulty of the underlying bug. That is what the word compositional in the title is doing: the benchmark is not a harder set of problems, it is the same problems presented the way people actually present them, with the presentation varied one factor at a time. Without the shared gold patch you could not tell a phrasing effect from a difficulty effect, and the whole exercise would collapse back into another leaderboard.

Across seven contemporary models, realistic inputs cut resolution rates by six point four percentage points on average, and can change the ranking between models. The controlled analysis is more actionable than that headline. Including the desired behaviour and the motivation significantly affects performance. Environment information and reproduction steps merely add tokens without measurable benefit. And linguistic style has only small, model-dependent effects, which is the result most likely to be assumed the other way round.

That gives advice a user can act on tonight. Say what you want to happen and why you want it, which most real prompts omit. Do not bother formalising your tone, and do not pad with environment details.

RealSWE. A Compositional Evaluation of Coding Agents under Realistic User Requests. By Gyuhyeong Kim and four others, from Sungkyunkwan University. Published on the thirty-first of August. This is the long version: the whole paper rather than the abstract.

Leaderboard scores on the SWE-bench family are the de facto standard for comparing coding agents. Those benchmarks harvest executable tasks from GitHub issues and their fixes, which makes them realistic in one sense and unrealistic in another. A GitHub issue describes a failure, provides reproduction steps and environment information, specifies the desired behaviour, and often suggests a solution. A user asking for the same thing says: this crashes on empty input, fix it. The underlying task is identical. The evidence supplied is not.

So benchmark performance may depend not only on task difficulty but on how the task is communicated, and that possibility has been raised before. Recent work has begun closing the gap with realistic benchmarks, evaluating prompts from real coding sessions, synthesizing underspecified tasks, or mutating GitHub issues to simulate user input. The authors identify two shortfalls in that line. Benchmarks grounded in genuine user data are closed-source and unavailable for independent inspection. Open alternatives approximate realism heuristically, truncating problem statements or injecting ambiguity, without systematic analysis or empirical grounding in real user data. So the existing work reveals that a gap exists while failing to say what causes it or how it should be measured.

The response is measurement before construction. The authors analyse user prompts from a large-scale dataset of real developer interactions with coding agents, and characterise each request along two orthogonal axes: the information it conveys, under a six-category taxonomy, and the language it is conveyed in, under four dimensions of style. Applying the same taxonomy to benchmark problem statements gives a like-for-like comparison, and the mismatch is stark on both axes. Requests carrying only a problem statement, alone or with limited extra context, are eighty-eight percent of real prompts and seven percent of benchmark problems. Eighty-seven percent of real prompts are casually written, against ninety-four percent of benchmark problems being formal.

Evaluated across seven contemporary models, realistic inputs reduce resolution rates by six point four percentage points on average, and can change the ranking between models. Every model declines, and the precise shape matters: this is not a benchmark that merely reshuffles a leaderboard, it is one where the whole field moves down together and the order changes underneath. Both halves are awkward. The absolute drop says the reported numbers overstate what these agents do for real users, and the reordering says the comparison between them was partly an artifact of how the tasks were written. The controlled analysis is where the actionable content is. Including the desired behaviour and the motivation significantly affects performance. Environment information and reproduction steps merely add tokens without measurable benefit. And linguistic style has only small, model-dependent effects.

The authors' limitations section is unusually practical, and it complicates their own advice. The implication is not that users should write benchmark-style issue reports for every request. It is narrower: state the desired behaviour when requesting a bug fix, and the motivation when requesting a feature. But they note that users may omit exactly this information because they do not recognise its value, or because the intended behaviour or motivation is itself unclear to them. Which is the real difficulty: the missing information is the part the user may not have. Their suggested responses put the work on the agent instead, either asking targeted clarification questions before implementing, or inferring the missing behaviour or motivation from the request and repository context and folding it into the task specification.

Two things to carry away, one for users and one for anyone building benchmarks. Practically, say what you want to happen and why you want it, and do not bother with the environment dump. Methodologically, this is a benchmark that measured the gap it was built to close before building anything, which is why its findings name specific fields of a taxonomy rather than reporting that realistic prompts are harder.