Original title: Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailable
Shai Vardi, João Sedoc · September 2026
People use AI models to help make organizational choices, but they often cannot tell if they should trust a specific recommendation. Most tools check overall model reliability or human trust instead of looking at one single answer. To solve this, researchers created epistemic warrant. It measures how stable an AI's preference is and how broadly that preference holds. They built a four-tier certificate that sorts choices into four groups: unstable, context-dependent, locally supported, and broadly supported. Tests showed this system matches expert rankings and aligns with crowd consensus. It also gives new information that goes beyond the model just stating its confidence, and it cannot be explained by simple task difficulty. This gives people a clear way to evaluate AI advice without ground truth.
Imagine you ask a friend whether to take route A or route B to work. You have no way to know if their preference is based on today's traffic or a general rule that applies every day. In this paper, the AI's recommendation is like that friend's advice, and the system checks how consistent and broadly applicable the preference is to decide if you should trust it.
Organizations using AI for decisions often can't verify if the advice is sound. This framework helps users decide when to trust a specific recommendation, reducing the risk of acting on unstable advice.
The abstract doesn't say this works for all types of decisions or AI models. It focuses on pairwise recommendations and uses specific validation methods, so it may not generalize to other contexts.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailable — Shai Vardi, João Sedoc
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.