Original title: Are Caption Metrics Broken? Latency, Deaf and Hard of Hearing User Ratings, and Bias across Technologies
Bernard Thompson, James Waller, Luz Fanny Calderon Torres, Lu Ming +4 · September 2026
Live captions on TV often have mistakes and timing problems, which makes it hard for deaf and hard-of-hearing viewers to follow dialog. The standard scores used to measure caption quality need to match what viewers actually experience. To test this, researchers showed volunteers live TV clips under four different caption setups: original broadcast captions with delays up to 7-12 seconds, broadcast captions synced to the sound, speech-recognition captions synced to the sound, and speech-recognition captions delayed by two seconds. Viewers rated caption quality and their own understanding. The researchers found that viewers rated broadcast and speech-recognition captions similarly. However, existing scoring metrics only correlated well with viewer ratings for human broadcast captions, not speech-recognition captions. Lastly, delays hurt the viewing experience, especially the 7-12-second broadcast delays.
Imagine you're watching a movie with subtitles, but the subtitles show up 10 seconds after the actor speaks. You can still read the words, but you've already moved on to the next scene, so you have to keep looking back and forth. Now imagine the subtitles are also slightly wrong—they say 'I'll meet you at the park' when the actor said 'I'll meet you at the bar.' You might still get the gist, but it's frustrating and you miss details. In this paper, researchers tested how well different ways of measuring subtitle accuracy match what real viewers actually experience, and found that the usual measurement tools work well for human-made subtitles but fail for automatic ones.
Deaf and hard-of-hearing viewers rely on live captions to follow TV, but current quality metrics were designed for human-generated captions. This study shows those metrics don't predict how well automatic captions actually work for real viewers, and that delays of 7–12 seconds—common on live TV—significantly hurt the experience. Without better metrics, caption quality could get worse as broadcasters switch to cheaper automatic systems.
The study only tested captions in English from U.S. TV broadcasts, so results may not generalize to other languages or regions. The ASR captions were generated by a single system, so different ASR technologies might yield different results. The survey used a specific set of 70 video clips, which may not represent all types of TV content. The abstract does not specify how participants were recruited or whether they represented the full diversity of the DHH community.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Are Caption Metrics Broken? Latency, Deaf and Hard of Hearing User Ratings, and Bias across Technologies — Bernard Thompson, James Waller, Luz Fanny Calderon Torres et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.