Original title: User Feedback Provides a Unique Signal that LLMs Can not Detect
Shachar Don-Yehiya, Leshem Choshen, Omri Abend · September 2026
Many researchers believe that everyday user feedback is too messy to help improve Large Language Models, which are AI systems trained on text. The authors tested this idea. They built synthetic test data with known correct answers, and they also gathered naturalistic, real-world data. Next, they had models revise their answers in two ways: one version used user feedback, and the other did not. Comparing the two showed that feedback-informed revisions fixed targeted mistakes at significantly higher rates. User feedback actually works very well. It only seemed useless because of broken evaluations. When an AI judge grades the revisions, it frequently fails to spot the genuinely corrected answer, systematically preferring the worse baseline version instead.
Imagine you're baking a cake and a friend tells you it needs more sugar. You add sugar and it tastes better, but when you ask a picky friend to taste it, they still say it's bad because they're focused on the original recipe. In this paper, user feedback is like the friend's advice that improves the AI, but the automated judge is like the picky friend who can't recognize the improvement and still prefers the original version.
Developers using automated judges to improve AI might discard useful feedback because the judges can't see the improvements. This slows down building better, more reliable AI systems that learn from real users.
The abstract doesn't explicitly state limitations, but the findings are based on specific evaluation paradigms and controlled comparisons; it doesn't propose a solution to the identified bias.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
User Feedback Provides a Unique Signal that LLMs Can not Detect — Shachar Don-Yehiya, Leshem Choshen, Omri Abend
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.