Original title: Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary, Maike Züfle +240 · September 2026
Language models are getting so good that standard translation tests are maxing out. Meanwhile, automatic scoring can be cheated, and human grading is hard to scale or repeat. This makes it tough to spot where models fail. To fix this, researchers created the Last Translation Benchmark. People wrote and reviewed difficult examples across text, images, audio, and video that intentionally break leading translation models. Each test item includes custom rules that detail exact failure cases, showing precisely how an answer went wrong instead of giving a vague score. The project is an ongoing, living test set that keeps accepting new examples over time, with the latest release, LTBv1, collecting accepted inputs up to September 1st 2026.
Imagine studying for a test where all the questions are easy, so you get a perfect score even if you don't understand the hard material. The test feels complete, but it doesn't reveal your weaknesses. Here, the benchmark is like a new test filled with tricky questions designed to catch mistakes. Then, the verification rules are like the answer key that points out exactly where you went wrong and why.
Standard translation tests are almost solved, so we can't tell if new models are actually better. This benchmark finds their blind spots, helping researchers know exactly where to improve translation technology.
The benchmark is limited to the examples contributed so far and focuses on identifying failures rather than improving translation models directly. It is a live dataset, so its coverage depends on ongoing submissions.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Last Translation Benchmark — Vilém Zouhar, Niyati Bafna, Mukund Choudhary et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.