Original title: Do speech foundation models really learn words?
Robin Huo, Ewan Dunbar · September 2026
Computer models for speech can tell spoken words apart. But researchers wondered: do these models actually understand a word as a single concept, or do they just hear its smaller speech sounds, called phonemes? To find out, the researchers analyzed two popular speech models, HuBERT and wav2vec 2.0. They used a mathematical method called residualization to remove the phoneme information from the models' internal signals. This left behind only what the models knew about the words beyond their basic sounds. They found that in their later layers, both models truly do learn separate representations for words that do not depend on local phonetic sounds. Separating these sounds also improved how well higher-level language details were captured during word discovery tasks.
Imagine you're learning to identify different types of fruit just by looking at them. At first, you might focus on the color and shape, which is like recognizing the sounds of words. But to truly know an apple from a pear, you need to understand their distinct identities, regardless of whether one is a slightly different shade of red or a bit rounder. In this paper, the researchers are trying to see if the AI models, after listening to lots of speech, can identify the 'fruit' (the word) based on its unique characteristics, not just its 'color and shape' (the sounds).
These models power many speech technologies, from dictation software to AI assistants. Understanding if they truly grasp words, not just sounds, helps us build more capable and reliable AI for everyone.
The abstract does not specify the exact fidelity of word representation or the precise gains in word discovery tasks. It also doesn't detail the specific downstream applications that benefit from this disentanglement beyond word discovery.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Do speech foundation models really learn words? — Robin Huo, Ewan Dunbar
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.