Erick Turner has published an important piece in CNS Drugs entitled “Publication Bias, with a Focus on Psychiatry: Causes and Solutions.” It should be required reading for any medical student, non-medical helping professional, or practicing prescriber. Rather than being an unbiased reflection of the underlying data, the psychiatric literature is instead sort of a funhouse mirror – the published results are consistently, overwhelmingly, positive – no matter what the actual data are. The implications are enormous. Note that the methodology in such trials is often deliberately constructed to bias such trials in favor of the sponsor’s drug – even so, publication strategies are used to minimize any bad news resulting from such trials, and to maximize the impression that the drug is effective.
Professionals that read these studies in the peer-reviewed literature (or systematic reviews based on these studies) will likely believe the results, and adopt biased impressions of psychiatric medications – they will believe them to be more effective, and less risky, than the actual data show them to be. When it comes time to inform their patients about treatment options, they will do their best to accurately inform their patients … but physicians exposed to biased literature are likely to then misinform their patients. The misinformation will almost inevitably be in the direction of lower thresholds for treatment and higher efficacy/lower risk for psychiatric medications.
Turner is correct in writing that these problems are in science across the board, but the potential for real-world harm is much higher in medicine in general and psychiatry in particular. There also seem to be different levels of awareness of these problems across scientific fields. I have had many conversations with social scientists who express awareness of the problems with publication bias, and are concerned – they often see the issues as a problem that must be solved over time. However, anecdotally I also hear of these same social scientists going to their prescriber and expressing concern about the evidence base in medicine and/or psychiatry – having read articles like Turner’s, and having some understanding of the challenges of making clinical decisions based on experimental evidence. Apparently, in many cases these conversations do not go well.
Having been trained in evidence-based medicine and in the primacy of randomized controlled trial data, many prescribers are apparently made very uncomfortable by the questioning of the paradigm. And one can easily imagine a clinician reading Turner’s article and then feeling completely overwhelmed: If Turner’s analysis is valid (I believe it is), the ramifications are so enormous that they may be difficult for the average clinician to believe. Human beings are justificationary creatures and physicians are no different. One strategy for addressing the problem would be to integrate this material into medical schools as soon as possible so that emerging physicians are well-trained on these issues. Of course, this is easier said than done, and could create significant angst among medical students as they try to reconcile a curriculum focused on clinical trial evidence with the unfortunate reality of selective and biased reporting.
So, what to do, moving forward? Turner focuses on problem-solving at the level of regulatory authorities, authors, and journals. His suggestions are all thoughtful, and most are realistic. Here I would like to close by repeating some more radical proposals that I believe would help solve the problem of publication bias (and associated problems) in psychiatric research. If academic medicine is actually interested in solving the deep scientific problems in the psychiatric literature, it could:
1. Admit that medical journals “have devolved into information laundering operations for the pharmaceutical industry” (Richard Horton, Lancet editor) and adopt a new mission for medical journals in which they only publish scientific criticism of trials (Richard Smith, BMJ editor).
3. Encourage dissent and transparency in journal publications. When publishing a clinical trial in an academic journal, provide patient-level data to a researcher with a dissenting point of view and allow them to publish a brief summary following the article. Deliberately subject the underlying data to a rigorous, public critique. Put any statistical tricks or changes in endpoints (which Turner explains well) on display and allow the reader to make their own decisions regarding the validity of the trial results.
4. Support efforts for data-sharing of unpublished patient-level data. Realize that sociologists and philosophers of science (and increasingly, the general public) will always be skeptical of any claims made where the underlying data are not available for re-analysis. Change scientific norms so that data unavailable for external re-analysis is simply not taken seriously.