In 2012 the biotechnology firm Amgen tried to reproduce 53 “landmark” studies in hematology and oncology. The company could only replicate six. Are doctors basing serious decisions about medical treatment on the rest? Consider the financial costs, too. A 2015 study estimated that American researchers spend $28 billion a year on irreproducible preclinical research.
As the Randall and Welser report emphasizes:
Incompetence and fraud together create a borderland of confusion in the sciences. Articles in prestigious journals appear to speak with authority on matters that only a small number of readers can assess critically. Non-specialists generally are left to trust that what purports to be a contribution to human knowledge has been scrutinized by capable people and found trustworthy.
The glorification of peer review by wide-eyed, incompetent journalists contributes to the junk science problem. The problem is probably worse in the social than in the physical sciences, but the report suggests that it has become increasingly worse in the physical sciences too.
Much research involves fishing for significant correlations that may be statistical artifacts and then playing them up. He who plays up best is most pleasing to the elite journals and is hence best at getting published in those journals.
Many years ago, with respect to the management field (related to my own field of industrial relations), Lex Donaldson wrote a book American Anti-Management Theories of Organization, in which he describes how the gamesmanship associated with the publication process had led to junk management theories. The Randall and Welser report is a broader discussion of the same problem.
Here are the first few of Randall and Welser's recommendations:
Much research involves fishing for significant correlations that may be statistical artifacts and then playing them up. He who plays up best is most pleasing to the elite journals and is hence best at getting published in those journals.
Many years ago, with respect to the management field (related to my own field of industrial relations), Lex Donaldson wrote a book American Anti-Management Theories of Organization, in which he describes how the gamesmanship associated with the publication process had led to junk management theories. The Randall and Welser report is a broader discussion of the same problem.
Here are the first few of Randall and Welser's recommendations:
1. Researchers should avoid regarding the p-value as a dispositive measure of evidence for or against a particular research hypothesis.
2. Researchers should adopt the best existing practice of the most rigorous sciences and define statistical significance as .01 rather than as .05.
3. In reporting their results, researchers should consider replacing either-or tests of statistical significance with confidence intervals that provide a range in which a variable’s true value most likely falls.
4. Researchers should make their data available for public inspection after publication of their results.
5. Researchers should experiment with born-open data—data archived in an open-access repository at the moment of its creation, and automatically time-stamped.
These recommendations are sensible to anyone who has done research in the social sciences, and I assume the same is true of the natural sciences.
These recommendations are sensible to anyone who has done research in the social sciences, and I assume the same is true of the natural sciences.
Astonishingly, tendentious left-wing bloggers (see Cory Doctorow's blog here and Michael Schulson's piece on Wired here) aim to turn these recommendations into a smear campaign against the National Association of Scholars.
Indeed, the reactions of tendentious "progressives" like Doctorow and Schulson offer evidence as to why university science has deteriorated in quality.
