Original title: Embedding qualitative data in LLM semantic space: A tutorial on conceptualization, measurement, and validation
Toshizumi Muta · September 2026
Researchers usually turn written survey answers into simple labels, which cannot measure how close two ideas are. Meanwhile, psychology measures closeness by checking how people's survey answers correlate. Today, large language models can turn any text into coordinates in a meaning-based space. If you calculate the angle, or cosine, between two coordinates, you get a score showing how similar the texts are. Researchers tested if this AI text closeness could replace real human survey correlations. They tested it across several established datasets, including personality traits, emotions in two languages, and personal values. They found that AI text analysis alone could not recreate human psychological patterns. Item text captured only part of the structure, failed entirely when relationships had positive or negative signs, and fell far short of human split-sample benchmarks. Plain wording does not capture what human respondents add. Still, text coordinates let researchers map complex answers without manual coding.
Imagine you're comparing two recipes. You look at their ingredient lists. If they share many ingredients, you guess the dishes taste similar. But that guess often fails: the amounts and cooking steps matter more than the list itself. In this paper, the researchers test whether a machine's measure of word overlap can replace the careful scoring of a personality questionnaire. The ingredient list is the words people use; the taste is the actual psychological trait they have.
If you analyze open-ended survey answers, you might hope AI could measure how similar people's feelings are. This paper shows that hope fails: machine text similarity does not match the structure of validated psychological scales. But it offers a new way to treat qualitative data as coordinates, useful for exploratory analysis without replacing standard measurement.
The test shows that linguistic structure alone does not reproduce psychometric structure; item texts recover only part, and not at all where a relation carries a sign. The results fall far short of a random split-sample ceiling. Also, meaning stays with the analyst, so the embeddings do not interpret themselves.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Embedding qualitative data in LLM semantic space: A tutorial on conceptualization, measurement, and validation — Toshizumi Muta — PsyArXiv — doi:10.31234/osf.io/6hxsw_v2
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.