Original title: Presumed Effective: The Manufacturing of an Evidence Base for AI-in-Education Through Flawed Meta-Analysis
Patrick Eugene O'Neill · August 2026
Many researchers claim artificial intelligence boosts learning, but are their conclusions reliable? Scientists often use meta-analysisA study that combines results from many smaller studies to get an overall answer., a mathematical tool that pools findings from many separate studies to measure an overall effect. In this paper, researchers audited 14 meta-analyses about AI in education to verify their quality. They checked how each paper picked its studies, combined ideas, and handled math. The results were alarming. None of the 14 reviews provided a valid basis for their claims. None properly defined the concept they tested, none adequately assessed publication biasThe tendency for studies with positive results to be published more often than those with negative results., and all showed severe variation among their combined studies. Furthermore, authors misapplied, miscalculated, and misinterpreted statistical tests. Most randomly checked primary studies were deeply flawed, usually because they measured outcomes that did not match the meta-analysis. Instead of genuine proof, these papers pooled biased results and presented them as solid facts, creating false certainty about AI tools.
Imagine a chef trying to prove that a new cooking technique makes all dishes better. She collects reviews from 14 different restaurants, but she doesn't check if each review actually used the technique, or if the restaurants only published positive reviews. She then averages the scores and declares the technique a success. In this paper, the meta-analyses are like that chef: they combined studies without checking if the studies measured the same thing, ignored that positive results are more likely to be published, and ended up with a false sense of certainty.
Policymakers are rushing to adopt AI in classrooms based on meta-analyses that claim strong evidence of effectiveness. This audit shows those meta-analyses are all fundamentally flawed—none have valid constructs, all ignore publication bias, and most primary studies don't match the claims. Decisions about billions of dollars and millions of students are being made on manufactured certainty.
The abstract doesn't say. It covers the audit of 14 meta-analyses and recommends stronger editorial standards, but does not discuss limitations of the audit itself.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Presumed Effective: The Manufacturing of an Evidence Base for AI-in-Education Through Flawed Meta-Analysis — Patrick Eugene O'Neill — EdArXiv — doi:10.35542/osf.io/n29zd_v3
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.