Original title: When Can We Work in Embedding Space? What Text Embeddings Preserve
Simon Freyaldenhoven · August 2026
Researchers often turn long text into text embeddings, which are compact lists of numbers that represent meaning. But do we lose important information when we replace the original words with these numbers? This paper tests that idea by modeling text as a blend of underlying topics. It shows that grouping these number lists gathers documents that share similar topic blends. It also shows that controlling for an embeddingA short list of numbers that stands in for a whole text, so a computer can compare documents by their numbers. works just like controlling for those hidden topics, meaning it is only valid if those topics capture the confoundingA hidden factor that affects both the thing you are studying and your outcome, making a relationship look real when it isn't. factors. In a test on 363 U.S. metropolitan areas, grouping embeddings of descriptions written by large language models revealed clear economic archetypes. These groups separated local job trends more sharply than traditional industry and demographic data.
You are packing a suitcase for a trip, and to save space you roll every outfit into a tight bundle. The problem is that two very different outfits can end up looking like nearly identical bundles, so later you cannot tell a business meeting from a beach day. In this paper a document is the suitcase and its compressed list of numbers is the rolled bundle; the hidden themes are the outfits. The work shows the compressed list is safe to use exactly when those hidden themes carry the information you care about.
Researchers who swap documents for embedding vectors in statistics now have a condition for when that swap is valid. The paper shows validity reduces to whether the hidden topic mixture captures the confounding, and demonstrates the payoff by separating local employment dynamics for 363 U.S. metropolitan areas from their economic descriptions.
The theory applies only under a generative model where documents are mixtures of latent topics, and validity always reduces to whether that mixture captures the confounding. The application is limited to LLM-generated economic descriptions of 363 U.S. metropolitan areas; the abstract offers no evidence for other text types, other tasks, or other data sources.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
When Can We Work in Embedding Space? What Text Embeddings Preserve — Simon Freyaldenhoven
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.