A new review of publicly available data shows that papers published in top psychology, economics, and general interest journals based on non-replicable studies are cited more than those that can replicate their results. Further, awareness of the failure to replicate appears to have no impact on citation rates and is rarely acknowledged in the citing publications. In other words, there appears to be no impetus to self-correct this trend within these fields.
The research was carried out by the economists Marta Serra-Garcia and Uri Gneezy from the University of California, San Diego, and published in the journal Science Advances. Serra-Garcia and Gneezy explain their findings:
âWhy are papers that failed to replicate cited more? A possible answer is that the review team may face a trade-off. Although they expect some results to be less robust than others, as shown in the predictions of experts, they are willing to accept this lower expected reliability of the results in some cases. As a result, when the paper is more interesting, the review team may apply lower standards regarding its reproducibility.â
A hallmark of good scientific research is replicability â the ability of other researchers, mimicking the studyâs experimental conditions, to produce the same results. While replicability is commonly required in hard sciences such as physics or chemistry, several critics have raised concerns about questionable practices within psychology concerning scientific standards of replicability. Serious doubts have been cast upon, for example, the replicability of research on depression.
As Pascal-Emmanuel Gobry wrote in 2016, widespread acquiescence to replicability failure suggests:
âThere is very good reason to believe that much scientific research published today is false, there is no good way to sort the wheat from the chaff, and, most importantly, that the way the system is designed ensures that this will continue being the case.â
In their new analysis, Serra-Garcia and Gneezy also determined, based on an analysis of journalsâ impact factors, that papers citing nonreplicable publications had similar impacts to those citing replicable publications. In terms of impact, then, there is no straightforward way to distinguish replicable from non-replicable research. This, together with the fact that non-replicable studies are more likely to be cited, arguably constitutes a âreplication crisisâ in the social sciences.
Why are papers that fail to replicate more often cited? The authors hypothesize that reviewers may accept lower reliability when a paper is more âinteresting.â This factor may be linked to the reviewersâ perception of a paperâs potential to create âhypeâ by using exaggerated or inaccurate claims regarding its findings.
Such studies âare more likely to receive media coverage and become famous ⌠this exposure may make the papers more likely to be cited. The effect of the hype lingers even after a study is discredited.â
****
Serra-Garcia, M., Gneezy, U. (2021). âNonreplicable publications are cited more than replicable ones.â Science Advances 2021;7. DOI: 10.1126/sciadv.abd1705 (Link)
This article reminds me of The Journal of Irreproducible Results which is supposed to be satire but unfortunately could be filled with much of pop science that makes it to catchy media stories.
Report comment
Well if they really want to, they can manipulate it all to make it look like “science”, of course to the general public.
The constant arguing about “science”, how psych is not science, is exactly where they want people.
They use that word “science” like they use the word “stigma”.
“hmm, how can we convince the public of this whole spiel”
Psych is about language and manipulation. Those things they understand quite well. They tend to label this kind of behaviour in people. Funny bunch
Report comment
Let me see what does this really say? Intelligence and wisdom have been abandoned for sensationalism and seduction. Basically, that academia is some far out ivory tower, far removed from society is a fallacy. In fact, it shows that the “good old school of hard knocks” or whatever you desire to call it is far more attuned not only to real life, but to real intelligence, wisdom, philosophy, science and art, etc. Academia, as we have known, is in crisis because it has had a total and complete breakdown. Although, there are those who would disagree, all I can say is the only truth I know is God. Thank you.
Report comment