Original title: The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations
Dmitrij Żatuchin · September 2026
If you ask an AI model for product recommendations several times, it will often give different answers because it picks words somewhat randomly. Researchers test this by asking the exact same question repeatedly, but they lack a clear rule for how many tests to run or how to measure consistency. To fix this, researchers introduced the Dice Roll Method. It models how these systems generate random text and breaks down why answers change, whether from random chance, wording, or software updates. They tested this method on five previous studies covering about 190,000 observations. They found that running 5 tries works for exploratory checks, 10 tries works for confirming patterns, and 15 tries gives rigorous results. Testing four different ways to score answers works better than relying on just one. Finally, they confirmed that researchers should run a small pilot test first to calculate the exact number of tries needed.
Imagine you're choosing a snack from a vending machine, but the machine sometimes gives you different items even if you press the same button. To figure out which snack you're most likely to get, you need to press the button multiple times and record what comes out. In this paper, researchers are pressing the button repeatedly on AI systems to see how consistent their brand recommendations are, and they've figured out the minimum number of presses needed to trust the results.
Companies and regulators need to check if AI systems give consistent advice. Without a standard method, audit results are unreliable and hard to compare. This protocol makes those checks consistent and trustworthy.
The abstract states that the fixed iteration tiers (n=5,10,15) do not transfer across all contexts, supporting a pilot-then-solve approach. The protocol is focused on brand recommendations and may not apply to all LLM uses. Validation reproduced reliability predictions but with no failures, which might limit generalizability.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations — Dmitrij Żatuchin
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.