Last Translation Benchmark

Last Translation Benchmark. By Vilém Zouhar and two hundred and forty-three others, from ETH Zurich, Johns Hopkins, and four other institutions. Published on the third of September. Its code repository is worked out: about seventy-five source files, with a readme and a licence.

Machine translation has a measurement problem on three fronts at once. Standard benchmarks are approaching saturation as models get stronger, so they no longer separate the leaders from each other. Automatic metrics are unreliable, vulnerable to reward-hacking, and give assessments you cannot act on. And gold human evaluation, the thing usually held up as the fallback, is not problem-free either: it often lacks reproducibility, objectivity and scalability. Taken together that leaves the field unable to track objective progress or identify where improvement should come from.

The response here is a benchmark defined by what breaks models rather than by what covers a language pair. The Last Translation Benchmark is a collection of human-authored and peer-reviewed examples that break leading machine translation models. Examples are not only text. They include images, audio and video, which matters because translation failures increasingly happen where the language is attached to something else.

The more unusual half is the evaluation method that comes with it. Each example arrives with handcrafted verification rules describing concrete failure cases on that specific example. That inverts the usual arrangement, where a single metric is applied across every item and tells you a score without telling you what went wrong. Here the checking is written per example, by hand, in advance, which is what makes future evaluation both reliable and actionable.

The benchmark is live rather than fixed. It accepts ongoing contributions, and the latest version is LTBv1, containing accepted contributions from before the first of September. Further releases are planned as new data continues to be collected.

The design bet is that a saturating benchmark cannot be rescued by a better metric, only by harder examples that come with their own definition of failure.

Last Translation Benchmark. By Vilém Zouhar and two hundred and forty-three others, from ETH Zurich, Johns Hopkins, and four other institutions. Published on the third of September. This is the long version: the whole paper rather than the abstract.

Machine translation is sometimes claimed to have reached lay human parity, and the public still perceives it as unreliable. The paper opens on that gap and offers an explanation for it that repays sitting with: one critical failure is enough to break a user's trust. So a system can be excellent on average and untrusted in practice, and an evaluation that reports averages will never see the problem.

Three things are broken at once. Static benchmarks are close to saturation, so they no longer expose the failure modes of strong models. Automatic metrics are unreliable, vulnerable to reward-hacking, and produce assessments you cannot act on, since a score does not tell you what went wrong. And gold human evaluation, the usual fallback, often lacks reproducibility, objectivity and scalability. Together those leave the field unable to track objective progress or identify where improvement should come from.

The construction is crowdsourced through a custom platform, and the pipeline a contributor follows is where the design lives. First they choose a source and target language, and these may be user-defined forms, dialects, regional variants or scripts, so Swiss German as spoken in Zurich is a valid target, as is Serbian in Cyrillic. They supply the input, which may be text, an image, audio or video, and they write a correct reference translation themselves. The platform then translates the input with up to ten models. The contributor inspects those outputs, identifies the common failure modes, and writes verification rules describing the concrete failures. A model then checks every candidate translation against every rule.

That pairing is then reinforced by human review. Each submission goes to a reviewer fluent in the relevant languages who checks the guidelines, and checks two things beyond them. That the example is fair, meaning an expert human translator given the same input could produce a passing translation. And that the errors the verification rules detect are perceptible and significant failures rather than technicalities. Here is a real example from the benchmark. A sentence in the Spanish of Colombia uses the word palistorneó, meaning that someone went crazy, and the word toche, which refers to a person and is specifically not derogatory. The rules encode exactly those two facts. Two of the ten models pass, and most of the failures are the same failure: they render toche as an insult.

The economics are reported and they are startling. Translating a thousand examples costs about twenty cents. Verification is the expensive half, because the prompt is longer, it runs against each rule separately with a reasoning model, and there are on average one point nine rules per example, so a thousand verifications cost about a dollar. On average a contributor made ten translation attempts before submitting a valid example. The average cost of one accepted example works out at twelve cents. The dataset is live rather than fixed: it accepts ongoing contributions, and the current release contains everything accepted before the first of September, with further releases planned.

The authors are careful about what this is. It is a stress test, not an estimate of typical translation quality, so the verification rules deliberately do not cover every aspect of quality, and the failure taxonomy is not exhaustive. They also separate out failures that are not really translation difficulty at all: models refusing on guardrails, models following an injected instruction inside the source text, and tokenization going wrong so the source is parsed as something else entirely. Those are tagged so they can be studied directly or set aside. And the honest structural limit is that as models improve, particular examples will be solved, so the taxonomy is expected to evolve to characterize whatever the frontier failures become next.

The design bet is that a saturating benchmark cannot be rescued by a better metric, only by harder examples that arrive carrying their own definition of failure. Writing the check by hand, per example, in advance, is what makes the evaluation both reliable and actionable, and it is why this had to be built by two hundred and forty-four people rather than by a pipeline.