Historical research guide · 0029 · 2005
Why Most Published Research Findings Are False
John Ioannidis
In one minute
John Ioannidis is an epidemiologist at Stanford, and this 2005 essay is the most-read article PLoS Medicine has published. His argument is arithmetic rather than complaint. Whether a published finding is actually true depends on four things: how large the study was, how much bias crept in, how many other teams are working the same question, and — above all — how likely the hypothesis was to be true before anybody tested it. Put realistic numbers into that model and, for most study designs in most fields, a claim that clears the usual significance threshold is more likely to be false than true. Ioannidis then lists the conditions that make it worse, and several are uncomfortable reading: small studies, small effects, many relationships tested with little preselection, flexible definitions and analyses, strong financial interests, and, surprisingly, a hot field with many teams racing one another.
Connection
Chapter 1 teaches readers to weigh a result rather than count results, and this is where the arithmetic comes from — a single published finding is a prior plus a measurement, not a verdict. It is also why chapter 13 keeps a ledger of what would settle each claim instead of a tally of papers.
From the source
The probability that a research claim is true depends on statistical power, the level of bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field.Abstract; Modeling the Framework for False Positive Findings
The smaller the studies conducted in a scientific field, and the smaller the effect sizes, the less likely the research findings are to be true.Corollaries 1 and 2
The greater the number and the lesser the selection of tested relationships, and the greater the flexibility in designs, definitions, outcomes and analytical modes, the less likely the research findings are to be true.Corollaries 3 and 4
The greater the financial and other interests and prejudices in a field, and the hotter the field in the sense of more scientific teams being involved, the less likely the research findings are to be true.Corollaries 5 and 6
Simulations show that for most study designs and settings it is more likely for a research claim to be false than true, so a single positive result crossing a significance threshold is not by itself strong evidence.Most Research Findings Are False for Most Research Designs and for Most Fields
Claimed research findings may often be simply accurate measures of the prevailing bias in a field; the improvements Ioannidis proposes are better-powered evidence, registration of hypotheses and study standards, and explicit attention to the pre-study odds rather than reliance on a single p-value.Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias; How Can We Improve the Situation?
The way in
https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124How to cite it
John Ioannidis (2005) Why Most Published Research Findings Are False doi:10.1371/journal.pmed.0020124