Original title: Lucky or Good? Outcome Noise, Effective Sample Size, and the Attribution of Skill
Karl T. Ulrich · July 2026
People often judge skill by looking at someone's past results. But results are often shaped by pure luck. The researchers wanted to know when past results actually reveal true skill, and what to do when they do not. They built a framework using two factors: the noise, or randomness, in each result, and the number of independent results you can observe over time. They found that major fields like venture capital, mutual fund management, and executive performance have too much noise and too few results to reliably judge an individual. Instead of judging these people solely on outcomes, evaluators should copy medicine: check whether a person follows practices that reliably produce better results across large populations.
Imagine you want to know which of two basketball players is better at free throws. You let each take ten shots. If the court is very windy and the ball wobbles randomly, the scores are mostly about the wind, not skill. Even if one sinks eight out of ten and the other only five, you can't be sure who is actually better—it could just be which gusts hit when. In this paper, the court noise is called outcome noise, and the ten shots are the effective sample size. The paper shows that many real-world situations—like picking a fund manager based on a few years of returns—are exactly like judging players on a very windy day with very few shots.
People get hired, fired, promoted, and funded based on their track records. This paper shows that in many high-stakes fields—mutual funds, venture capital, executive leadership—the noise in outcomes is so high that individual performance numbers can't reliably separate skill from luck. If we keep acting like they can, we reward randomness and punish competence.
The abstract does not provide specific data or examples to quantify how much noise or effective sample size exists in each domain. It also does not explain how to implement the proposed population-level validation approach in practice—for instance, how to identify which practices are truly associated with better outcomes, or how to avoid the same noise problem at the population level. The framework is theoretical and prescriptive but not tested empirically in the abstract.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Lucky or Good? Outcome Noise, Effective Sample Size, and the Attribution of Skill — Karl T. Ulrich
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.