Original title: Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning
Kevin Du, Alexander Hoyle, Laura Ruis, Acyr Locatelli · September 2026
When AI models write out step-by-step reasoning, people assume we can read those thoughts to see how the AI made up its mind. Many tools use another AI to judge these steps, check for mistakes, and give feedback. But does the written text really show which steps actually mattered? To test this, researchers defined a step's true importance by calculating how much adding that step changes the chance of getting the right final answer. They ran many trial completions to measure this effect. Then, they tested whether AI judges could spot these important steps. They found capable AI judges beat a simple prevalence baseline, but they still fall far short of the noise ceiling. Fine-tuning helps spot key steps in wrong answers, but stays far from the ceiling on correct answers. What looks readable is not truly interpretable.
Imagine following a recipe where each step is described in detail. Sometimes a step sounds important because it's long or uses fancy words, but actually, a short, simple step might be the key to the dish turning out well. In this paper, the researchers found that even when an AI reads the reasoning steps of a model, it can't always tell which steps are truly critical for getting the right answer, because the importance isn't always obvious from the words used.
If we trust the written reasoning steps to judge a model's decisions, we might misdiagnose errors or mis-train the system. In high-stakes fields like medicine or law, this could lead to incorrect assessments of model reliability.
The abstract doesn't specify the models or tasks used, so it's unclear how general the findings are. The study defines importance using a specific estimation method, Monte Carlo rollouts, which may not capture all aspects of importance. Also, the evaluation is limited to language model judges, so other methods of assessment might yield different results.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning — Kevin Du, Alexander Hoyle, Laura Ruis et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.