The p-value: How a Convenient Threshold Became Science's Gatekeeper

In 1925, Ronald Fisher published "Statistical Methods for Research Workers," a practical manual for scientists who needed statistics but lacked mathematical training. In a passage about tests of significance, he wrote that a result could "conveniently" be considered significant if the probability of observing it by chance, the p-value, was less than 0.05. He chose this threshold informally, acknowledging it was arbitrary. He did not say that p < 0.05 meant a result was true, or that p > 0.05 meant it was false. He described it as a useful preliminary filter. That nuance has been largely lost ever since.

Fisher was one of the most consequential and most difficult figures in 20th-century science. He invented analysis of variance (ANOVA), the F-test, maximum likelihood estimation, the design of experiments with randomisation and blocking, and the foundations of population genetics that reconciled Darwin with Mendel. He was also deeply racist, and his eugenics work is now widely condemned. In his final years he denied the statistical evidence linking smoking to lung cancer while receiving funding from tobacco companies. The man who built modern statistics also provided cover for one of history's most dishonest public health campaigns.

The p < 0.05 threshold spread because journals found it useful. Publication required significance, and significance required p < 0.05. This created an immediate and systematic incentive for practices later called p-hacking: collecting data until p < 0.05 appeared, running multiple analyses and reporting only the significant one, dropping observations that pushed results above the threshold. These practices were rarely deliberate fraud: researchers genuinely believed they were doing science. They were doing science through a broken filter.

The replication crisis of the 2010s made the problem undeniable. In 2015, the Open Science Collaboration attempted to reproduce 100 published psychology experiments and succeeded in replicating approximately 36%. In medicine, John Ioannidis had already published "Why Most Published Research Findings Are False" in 2005. The American Statistical Association issued a formal statement in 2016 warning against using p < 0.05 as a binary decision threshold. Major journals including Basic and Applied Social Psychology banned null hypothesis significance testing entirely.

The honest state of statistics in 2025 is that p-values are not being replaced but recontextualised. Effect sizes, confidence intervals, Bayesian credible intervals, pre-registration of hypotheses, and large replication samples are increasingly required alongside them. In machine learning, the p-value plays little direct role: model performance is evaluated on held-out test sets. But the underlying problem Fisher's threshold illuminated, that evidence is continuous and decisions are discontinuous, and any threshold mapping one to the other will produce pathological behaviour, remains the central problem of applied statistics, with no clean solution.