Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev
Nebius
Published November 6, 2026 · on the feed November 10, 2026 · ▲ 21 · arXiv 2611.10304
The abstract
Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver--judge--repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.
The Brief
1:51A short podcast summary of the paper. Written and voiced by AI.
Transcript
Longtail-SWE. High-Quality SWE Tasks for Underrepresented Languages. By Vladimir Dobrovolskii and two others, from Nebius. Published on the sixth of November.
For many programming language ecosystems, challenging repository-level benchmarks are still in short supply. Part of the reason is the filtering. When candidate tasks are scarce to begin with, throwing out every one with an ambiguous description or a defective test can discard engineering problems that were genuinely valuable. Strict filtering keeps a benchmark clean, but in these ecosystems it also keeps it small.
Longtail-SWE is a benchmark of two hundred and two issue-resolution tasks, spread across thirteen languages that existing repository-level benchmarks cover only thinly. They are Clojure, Dart, Elixir, Erlang, F sharp, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift.
Instead of discarding flawed candidates, construction repairs them, through a pipeline of solver, judge and repairer. A solver attempts each candidate task. A judge then reviews the task alongside the solver's trajectory and the test results, and uses that review to guide revisions to the tests and to the task description. The revisions target two kinds of faulty test: one that rejects a valid alternative implementation, and one that accepts an incorrect solution. Throughout, the aim is to keep the hard tasks hard and to preserve what each task was meant to require.
Nine model configurations are evaluated, frontier models among them. The analysis looks at task-solving performance, at consistency across repeated attempts, and at resource usage. The abstract does not give the scores themselves, so what can be said here is what was measured rather than how the models fared.
A small benchmark in a language few benchmarks reach is only as good as its tests, and the repair loop is the answer to that. Alongside the benchmark instances come executable environments, the agent trajectories, and an extra set of easier tasks meant for evaluating smaller models.
Paper Quest · 7 chapters · about 36 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Some languages barely appear in coding benchmarks. Here is why that is hard to fix.
You want to test coding agents on real issues in languages like Haskell, Erlang or Lua. Candidate tasks from real pull requests are scarce in these ecosystems.
The road ahead
- 1 · The problem ~6 min In rare programming languages, good test tasks for coding agents are scarce, and many candidates have vague descriptions or faulty tests.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Instead of discarding flawed candidate tasks, a solver tries each one, a judge diagnoses the defects, and a repairer fixes tests and descriptions.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each candidate is attempted by a solver, reviewed by judges, repaired in a loop, and the finished tasks are used to evaluate nine model configurations.
- 6 · What they found ~3 min
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.