Original title: Phoneme overlap analysis of the visual benefit for speech-in-noise perception
John Magnotti, Yue Zhang, Lin Zhu, Yingjia Yu +1 · September 2026
Watching a speaker's face helps you understand them in noisy rooms. Scientists call this boost the visual benefit. Usually, researchers score a listener's answer as totally right or totally wrong. This misses smaller clues that listeners actually catch. To fix this, researchers tested a method based on phonemes, which are the distinct sounds that build words. Instead of grading whole words, they counted the matching sounds between what was said and what listeners heard. Testing people on heard and seen words revealed that looking at shared sounds is much more sensitive. It detected visual help in over twice as many trials. The boost changed massively across different words and sounds, with some sounds helping far more than others. Researchers could even predict the visual benefit for untested words.
You are trying to hear a song through static on the radio. Normally you just say whether you got the whole song right or wrong. But some notes are easier to pick out than others. Here, instead of checking whole words, the researchers check each individual sound, and find that some sounds like 'f' and 'th' are much easier to hear when you see the speaker's face.
For people with hearing loss, knowing exactly which sounds are hardest to lip-read could improve training and device design. Clinicians and researchers can now measure the visual benefit at the level of individual sounds, not just whole words. This finer detail also helps engineers build better audiovisual speech recognition systems.
The abstract doesn't say. It reports results from 55 participants with specific English words, so generalizability to other languages or populations is not established. The prediction of untested words has a median error of 9%, so it is not perfectly accurate.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Phoneme overlap analysis of the visual benefit for speech-in-noise perception — John Magnotti, Yue Zhang, Lin Zhu et al. — PsyArXiv — doi:10.31234/osf.io/s5wbt_v1
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.