Original title: Investigation of Word Recognition Methods for Japanese Fingerspelling Using Large Language Models
Yousun Kang, Renta Kuzuyama, Naoto Tsuta, Ryota Murai +1 · September 2026
Reading fingerspellingSpelling words using hand shapes, one letter at a time. from standard video is hard for computers. Fast hand movements cause motion blurBlurring caused by fast movement that makes details unclear in a video frame., and fingers often block each other from view. This causes tracking tools to make frequent mistakes. To fix this, researchers tested a system that connects video processing with a Large Language Model, or LLM, which is an artificial intelligence trained on vast amounts of text. The visual system tracks 3D hand joints from a single regular video and spots motion patterns. Then, the LLM uses its built-in word knowledge to guess the correct words whenever the video signals are messy or incomplete. By itself, the visual tracker scored only an 11.66% word accuracy on continuous signing. But when the researchers fed the visual data into the LLM, accuracy jumped to a high of 87.73%. Passing deeper visual features directly into the model worked best to prevent character errors.
Imagine trying to read a friend's lips across a noisy room. You catch most of the shapes, but some are blurred or hidden by their hand. You fill in the gaps by guessing what they would logically say next. Here, the camera captures blurry hand motions, and a computer program that knows common word patterns fills in the missing letters.
For deaf and hard-of-hearing individuals, fingerspelling recognition without uncomfortable sensors could make everyday communication easier. Current vision-based systems fail due to motion blur, but this work shows that combining video with language models can overcome that. While still experimental, it points toward practical, non-invasive assistive technology.
The results are from a closed-set of 28 words under controlled conditions, so real-world performance with larger vocabularies and natural signing may differ. The visual encoder alone achieved only 11.66% accuracy, showing heavy reliance on the LLM. The study does not propose a new LLM architecture and only compares integration strategies.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Investigation of Word Recognition Methods for Japanese Fingerspelling Using Large Language Models — Yousun Kang, Renta Kuzuyama, Naoto Tsuta et al. — Sensors — doi:10.3390/s26196044
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper.