Original title: From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs
Urja Pawar, Rajitha Ramanayake, Owen O'Neill, Nabeel Kemal +3 · September 2026
Large language models often make up false facts, known as hallucinations. Catching these errors without trusted reference documents is hard. Missing a lie causes harm, but raising false alarms wastes human review time. Researchers studied two clues available from model outputs. One is semantic entropy, which measures whether different generated answers disagree in meaning. The other is token uncertainty, which tracks the model's confidence scores for individual words. These clues cover each other's blind spots. Semantic entropy fails when repeated answers agree on the same mistake, while token confidence can miss consistently confident errors. The researchers built methods to combine these signals, including TopK to pool word scores across answers, and Stacked to learn from both signals together. They found that no single tool is universally strongest, but Stacked performed best in nearly half of the tested cases, while unsupervised approaches remained competitive.
At a crosswalk, you rely on both the walk signal and your glance at oncoming cars to decide when to cross. Sometimes the walk signal is on, but a distracted driver is approaching, and sometimes you miss a car because it’s quiet. In this paper, detecting AI errors is similar: one method checks if different answers agree in meaning, and another checks how sure the AI is about its words. Using both gives a better safety check, but neither catches every mistake.
In high-stakes AI uses, missed errors harm users and institutions, while false alarms waste human time. Better detection methods lead to safer systems and more efficient oversight.
The methods are not universally effective, as no single approach works best across all cases. Performance depends on careful threshold calibration, and the study is limited to the benchmarks and language models tested.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs — Urja Pawar, Rajitha Ramanayake, Owen O'Neill et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.