Original title: Systematic Evaluation of Nation-State Propaganda in LLM Outputs
Kevin T. Greene, Jacob N Shapiro · September 2026
People worry that AI chatbots might repeat propaganda from state-run influence campaigns. Past studies tested this using tiny sets of questions with unclear methods. To fix this, researchers built an automated, repeatable testing system. It takes real government propaganda and news stories, creates test questions from them, and checks the answers five well-known AI models give. The system checked if models copied Kremlin talking points, shared those views without adding corrective facts, or cited propaganda sites. They found that chatbots repeated propaganda and cited propaganda sources far less often than earlier small studies suggested. In fact, these risks were especially rare when questions looked like regular web searches about Ukraine. The team can now use this setup to monitor AI responses over time.
Imagine you're a teacher checking if students are copying from a banned website. You only ask a few questions to a few students, and you don't track which sites they visit. You might think cheating is everywhere. But if you ask many questions to all students and log every site they open, you may find that students rarely copy from banned sites, especially when the question is simple. Here, the researchers built an automated system that asks many questions from actual government talking points and checks the responses of five different chatbots. They found these bots repeat government messaging much less often than earlier small tests suggested, especially for ordinary factual questions.
If you ask a chatbot about Russia-Ukraine, you might worry it will repeat Kremlin talking points. This paper finds such risks are lower than earlier small-scale tests claimed — at least for ordinary searches. The automated monitoring system they built can keep checking as models and propaganda evolve.
Risks may still be higher for less common queries or other propaganda sources not tested. The study only examines Kremlin narratives about Ukraine; other state-backed campaigns (e.g., Chinese or Iranian) are not covered. The five models tested may not represent all chatbots. The automated question generation might not capture every way propaganda could appear. The results show lower rates, not zero, and don't guarantee safety in all contexts.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Systematic Evaluation of Nation-State Propaganda in LLM Outputs — Kevin T. Greene, Jacob N Shapiro — SocArXiv — doi:10.31235/osf.io/57buw_v2
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.