Even when user data is hidden, an LLM's decisions can reveal private information.

Personalized AI agents often compress what they remember about you into hidden user models. People assume these internal models protect privacy because the AI removes your exact words, stopping attackers from directly stealing old text. However, the researchers found a major problem: the choices an AI makes still reveal your private details. To show this, they created UMPeek, a black-box attack that reconstructs your hidden profile just by chatting with the agent. It works by making guesses based on choices left open by a prompt, switching between normal follow-up tasks, and keeping only the ideas that the AI's behavior supports and does not contradict. In tests and real-world systems, UMPeek beat existing attacks and still stole private data even when defenses tried to clean up the responses. Hiding backend records does not keep you safe if memory guides behavior.

Why it matters

People using AI assistants that remember their preferences might think their data is safe if hidden, but this research shows that the AI's choices alone can leak sensitive details. This matters for anyone relying on personalized AI in healthcare, finance, or daily life, where privacy is critical.

Questions answered