Original title: How much selection would be enough? Bounding the learning claim of El Salvador's artificial intelligence tutoring pilot
Jorge Aníbal Restrepo-Morales, Eduar Antonio Rodríguez Flores, Emerson Andrés Giraldo Betancur, Freddy Zea Restrepo · September 2026
A report claimed that students in an El Salvador pilot using artificial intelligence tutors matched the 2022 test averages of Germany and Sweden. Based on this, the country plans to expand the program to all public schools. But did the artificial intelligence actually cause this leap, or did officials just test their highest-performing students? Researchers studied how much student selection could explain these scores if the tool produced zero learning. They took El Salvador's overall test score distribution from national data. Then, they calculated how small of an elite top group you would need to pick to hit Germany's average scores. You only need the top 5.5% of students in mathematics, 11.7% in science, and 17.2% in reading. Because the pilot tested only about seven students per school, selecting top students is plausible. Without selection, the claim would mean an unrealistic gain of about 6.5 years of schooling in one year.
Imagine a school claims its new breakfast cereal makes students run faster. They test 7 students from each class and report the average time matches Olympic sprinters. But if they only tested the fastest 5% of students in each class, the average would look Olympic even if the cereal did nothing. In this paper, the researchers calculate exactly how few students you would need to pick to fake the reported test scores—and show the pilot tested so few per school that such picking is easy.
Governments worldwide are spending billions on AI tutoring, often citing pilot results as proof. If a reported gain can be reproduced by cherry-picking a tiny fraction of students—without any actual learning—then policy decisions rest on an illusion. This paper shows why every such claim must come with baseline data and a sampling protocol, or it is worthless for deciding how to educate millions of children.
The analysis assumes the national PISA 2025 distribution accurately represents the pilot schools' baseline, which may not hold if those schools differ systematically. It also assumes a normal distribution shape; the authors note robustness checks for skewness but cannot fully rule out other distributions. The paper does not test whether the pilot actually worked—it only bounds how much selection could explain the result.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
How much selection would be enough? Bounding the learning claim of El Salvador's artificial intelligence tutoring pilot — Jorge Aníbal Restrepo-Morales, Eduar Antonio Rodríguez Flores, Emerson Andrés Giraldo Betancur et al. — EdArXiv — doi:10.35542/osf.io/6gj9u_v1
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.