Original title: A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval
Santiago Poveda-Gutiérrez, Hideki Nakayama, Mayumi Bono · September 2026
Most sign language systems treat recognition like a closed multiple-choice quiz. They map hand movements directly to fixed labels called glosses, meaning they cannot recognize any sign they were never trained on. To fix this, researchers built a reverse sign language dictionary that works without gloss training and handles an open vocabulary. Instead of guessing a label, an open-weight vision-language computer model watches a video clip of continuous signing and writes a procedural descriptionA step-by-step explanation of how a sign is performed with hands and body. detailing the physical hand and body movements. Next, a sentence encoder matches that written description to the closest entry in a dictionary of target descriptions. Testing on Japanese Sign Language clips, training the video captioner boosted top-10 accuracy on familiar signs from 4.5% to 49%. Crucially, accuracy on completely unseen signs jumped from 11.5% to 21.0%, a task traditional fixed-label classifiers cannot attempt at all.
Imagine you're at a foreign market trying to buy fruit, but you don't know the names. You point and describe the fruit's shape and color to the vendor. If your description is vague, you might get the wrong fruit. In this paper, the system describes sign language gestures in words and looks up those descriptions in a dictionary to find the meaning, similar to how you'd describe fruit to identify it.
This could help deaf people and sign language users interact with technology that understands a wider range of signs, including those not in standard dictionaries. It makes sign language recognition more flexible and accessible for everyday use.
The system's performance is limited by captioning quality, which needs improvement. It is tested only on Japanese Sign Language and a specific dialogue corpus, so generalizability to other sign languages is not established. The approach relies on accurate procedural descriptions, which may not be available for all signs.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval — Santiago Poveda-Gutiérrez, Hideki Nakayama, Mayumi Bono
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.