Original title: CHARM: Character Hallucination for Multicultural Role Play Benchmark
Sunkyung Han, Nahyeon Park, Gaeun Seo, Seunghyun Yoon +1 · September 2026
When AI plays a character, it should only know what that character would know. But models often cheat and use modern or outside knowledge they should not have. Researchers built CHARM, a test with 40 real and fictional figures across five cultural regions. Native reviewers checked this test. The researchers tested the AI in two steps. First, they checked if the model realized a question fell outside the character's world. Second, they checked if it actually refused to answer. They found that character errors happen mostly because models fail to hold back. Even when an AI explicitly admits its character cannot know a fact, it blabs the answer anyway. The AI simply fails to suppress facts stored in its memory. These failures also change depending on the character's home culture.
Imagine you're playing a friend from the past in a board game. You know what that person wouldn't know about modern events, but when someone asks, you accidentally give the modern answer anyway. In this paper, AI characters face similar tests where they must stick to their own knowledge limits, and often fail even when they recognize the question doesn't fit.
This matters because role-playing AI that ignores cultural boundaries can spread misinformation or disrespect traditions. Users engaging with these models deserve accurate, culturally sensitive interactions.
The abstract doesn't state explicit limitations, but it only includes characters from five cultural-linguistic regions and tests six LLMs, so the findings may not generalize broadly to all AI systems or cultural contexts.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
CHARM: Character Hallucination for Multicultural Role Play Benchmark — Sunkyung Han, Nahyeon Park, Gaeun Seo et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.