Correcting text measurement error alone can still mislead causal conclusions.

Researchers often want to find cause-and-effect relationships using written text, like personal notes, instead of simple survey numbers. To do this, they use computer programs to read the words and label them. However, these text-labeling tools often make mistakes, creating measurement error. At the same time, real-world data suffers from missing factors, known as omitted confounders, which can also distort the results. Fixing measurement mistakes alone does not solve the problem. The researchers derived mathematical formulas to calculate the exact bias that occurs when these problems are ignored or only partly fixed. Using these formulas, they built a new system to estimate the Average Treatment Effect of a text-based factor. They demonstrated their approach on text from the Midlife in the United States study to examine how social interaction affects cardiovascular health.

Why it matters

Researchers using text data to study causes, like the effect of social interaction on health, often get biased results because NLP labels are imperfect and other important factors are missing. This work shows how to handle both problems together, making conclusions more trustworthy. It matters for any field that relies on text for causal inference, from social science to medicine.

Questions answered