Original title: How Replicable Are Statistically Significant Findings?
Patrick Vu, Stefan Faridani · August 2026
Scientists often treat a result as real if it passes a statistical cutoff, called a significance threshold. But do these findings hold up if another team runs the same test? Researchers wanted to see if results that barely pass these thresholds stay significant in new studies of the same size. They looked at published work in experimental economics, psychology, and social science. They calculated how likely a study is to repeat successfully based on its p-valueA number that tells you how likely it is that your result happened by chance alone; lower means less likely due to chance., a number that measures statistical strength. They found that a finding with a p-value of 0.05 repeats only 0.10 to 0.25 of the time. This happens because original studies have low statistical powerThe ability of a study to detect a true effect; low power means the study is less likely to find an effect even if it exists., not publication biasThe tendency for journals to publish only results that are statistically significant, which can distort the scientific record.. One study offers only suggestive evidence.
You are trying to decide if a new study technique helps you remember facts. You try it once and your test score jumps from 70% to 85%. That seems like a big improvement. But when you try the same technique again, your score only goes up to 75%. The first result was partly due to luck. In this paper, the researchers show that many scientific findings that just barely meet the standard for being considered real are like that first test score—they often fail to show the same effect when the experiment is repeated.
Researchers and policymakers often treat a single statistically significant study as evidence. This paper shows that a finding with p=0.05 has only a 10-25% chance of being confirmed in a replication. That means many published results may be false positives, wasting resources and misleading decisions. Stronger conclusions require multiple studies.
The analysis is limited to studies in experimental economics, psychology, and social science, and to replications with the same sample size. The estimates may not generalize to other fields or replication designs. The nonparametric estimator may rely on assumptions not stated in the abstract.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
How Replicable Are Statistically Significant Findings? — Patrick Vu, Stefan Faridani
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.