LLM personas that hold the right opinions can still fail to revise them like humans.

Researchers want to use AI personas to simulate human public opinion. Today, tests only check if an AI holds realistic beliefs at a single moment. But real people also change their minds when they learn new facts. The researchers wanted to see if AI models update their beliefs like humans do. They created a diagnostic test that gives identical, balanced information to both humans and AI personas, then compares how their views shift. Every model they tested failed. Some models overreacted, changing their opinions five to seven times more than real people. Another barely changed its views at all. One model even did the exact opposite of humans: it became more hostile toward opposing political groups after reading balanced facts. Instead of genuinely thinking through new arguments, the AI models stuck to internal stereotypes about how those personas should act.

Why it matters

Pollsters and researchers are starting to use LLMs as stand-ins for real people in surveys. If the models only get the static opinions right but update beliefs in a fundamentally inhuman way—becoming more hostile after balanced information, shifting too much, or not at all—then conclusions about how public opinion changes under new arguments could be deeply misleading. This framework gives them a test to catch that failure before it affects a study.

Questions answered