Original title: Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks
Oline Ranum, Edward Fish, Simon Hadfield, Richard Bowden · September 2026
Computers struggle to translate sign language accurately. Today, researchers test these systems using a common score called BLEU-4A translation quality metric that counts matching word sequences between machine output and human references.. But this score can be fooled. Models can guess words using basic language patterns instead of truly understanding the signs. To test this, researchers checked six translation models across two datasets. They found higher BLEU-4 scores do not mean the system actually understands sign language better. The team built a new testing method based on language-learning exams. It uses a question-and-answer test run by an open-weight large language model to see if vital facts survive translation. This tool matches human judgment better and handles reworded text six to seven times better than BLEU-4, revealing true differences between systems that older scores missed.
Imagine grading a student's essay by only counting how many words match a textbook, not if they understood the topic. The student could copy phrases perfectly without grasping the concepts, leading to a high grade but poor learning. In this paper, BLEU-4 is like counting matched words, while the new QA protocol is like asking the student questions to see if they can explain the main ideas in their own words.
If the standard metric for evaluating sign language translation is flawed, we might overestimate how well technology communicates with deaf users. A better evaluation method helps ensure translations actually convey meaning.
The evaluation is based on two specific datasets, Phoenix-2014T and CSL-Daily, so it may not generalize to all sign languages or contexts. The alternative metric is proposed but not yet widely adopted or tested across more scenarios.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks — Oline Ranum, Edward Fish, Simon Hadfield et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.