A recent study, published open-access in Psychological Medicine by a team of researchers out of the Netherlands and the UK, analyzes the cumulative impact that publication biases have on the apparent efficacy of antidepressants and psychotherapy for the treatment of depression. The results indicate the necessity for clinicians across medical fields to be aware of the possibility of biased research results and distorted claims of treatment efficacy.
“The problem of study publication bias is well-known,” the researchers, led by Dr. Ymkje Anna de Vries from the University of Groningen, write. “Our examination of antidepressant trials, however, shows the pernicious cumulative effect of additional reporting and citation biases, which together eliminated most negative results from the antidepressant literature and left the few published negative results difficult to discover…We have already shown that similar processes, though more difficult to assess, occur within the psychotherapy literature, and it seems likely that the effect of these biases accumulates whenever they are present.”
While evidence-based medicine seeks to be the foundation of clinical practice, it depends on the quality of the evidence and clinician awareness of the nuances within the evidence. The authors note, “up to half of all randomized controlled trials (RCTs) have never been published, and trials with statistically significant findings are more likely to be published than those without,” adding, “negative trials face additional hurdles beyond study publication bias that can result in the disappearance of non-significant results.”
Financial motives for the pharmaceutical industry to restrict the dissemination of unfavorable results is one possibility for skewed representation of negative studies. Still, biases are also present in psychotherapy research. Another explanation for this trend involves the pressures for researchers to produce positive results, as journals with higher visibility publish more positive studies than negative studies.
The authors of this study delineate four major biases:
- Study publication bias: “the non-publication of an entire study.”
- Outcome reporting bias: “the non-publication of negative outcomes within a published article or to switching the status of (non-significant) primary and (significant) secondary outcomes.”
- Spin: “Reporting strategies that could distort the interpretation of results and mislead readers…e.g., instead of concluding that treatment X was not more effective than placebo, concluding that treatment X was well tolerated and was effective in patients who had not received prior therapy.”
- Citation bias: “an obstacle to ensuring that negative findings are discoverable. Studies with positive results receive more citations than negative studies, leading to heightened visibility of positive results.”
To understand the impact of these biases, the researchers assessed the evidence base for antidepressants by identifying 105 depression trials, 74 from a previous study on publication bias and 31 from the Food and Drug Administration (FDA) database, where experiments with non-significant results are accessible, even if unpublished.
Figure 1 illustrates the cumulative impact of study publication bias, outcome reporting bias, spin, and citation bias. 50% (53) of trials were considered positive by the FDA, and 50% (52) were deemed to be negative or questionable. All but one (98%) of the positive studies were published while less than half (48%) of the negative trials were published. In total, 25 were negative of the 77 trials that were published.
10 of the 25 negative trials became positive through either omitting unfavorable outcomes or switching the status of primary and secondary outcomes.
Additionally, 5 of the remaining 15 negative articles were published with a spin in the abstract. For example, “one article reported non-significant results for the primary outcome, yet concluded that the trial ‘demonstrates an antidepressant effect for fluoxetine that is significantly more marked than the effect produced by placebo.’” Similarly
“Only four of 77 published trials unambiguously reported that the treatment was not more effective than placebo in that particular trial,” the study reports. “Compounding the problem, positive trials were cited three times as frequently as negative trials.”
, de Vries and colleagues found that psychotherapy trials “clearly provide a more positive impression of the effectiveness of psychotherapy than is justified by available evidence.” They find that positive psychotherapy trials are cited twice as often as negative trials. However, the researchers found psychotherapy trials more taxing to detect due to the lack of a standardized trial registry. Furthermore, small sample sizes warrant more evidence to make definitive conclusions concerning psychotherapy publication bias.
The researchers offer several approaches to preventing biases:
- Include a regulatory requirement to enforce the mandatory prospective registration for study publication.
- Incorporate behavioral interventions (e.g., psychotherapy) and phase 1 trials (healthy volunteer) within those requirements (until now, they are excluded).
- Closely examine registries by employing independent researchers.
- Publish study protocols, or “registered reports,” in which a journal accepts a study based on the introduction and methods, sans results.
- Encourage peer reviewers to interpret a study’s actual results and check they are accurately reported in abstracts, and that they promote the inclusion of negative study results.
De Vries et al. conclude with practical implications, in which clinicians “must be aware of the potential for bias to distort apparent treatment efficacy, which poses a threat to the practice of evidence-based medicine.”
****
De Vries, Y. A., Roest, A.M., de Jonge, P., Cuijpers, P., Munafò, M.R., Bastiaansen, J.A. (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine 1–3. https://doi.org/10.1017/S0033291718001873 (Link)
