A recent study, published open-access in Psychological Medicine by a team of researchers out of the Netherlands and the UK, analyzes the cumulative impact that publication biases have on the apparent efficacy of antidepressants and psychotherapy for the treatment of depression. The results indicate the necessity for clinicians across medical fields to be aware of the possibility of biased research results and distorted claims of treatment efficacy.
“The problem of study publication bias is well-known,” the researchers, led by Dr. Ymkje Anna de Vries from the University of Groningen, write. “Our examination of antidepressant trials, however, shows the pernicious cumulative effect of additional reporting and citation biases, which together eliminated most negative results from the antidepressant literature and left the few published negative results difficult to discover…We have already shown that similar processes, though more difficult to assess, occur within the psychotherapy literature, and it seems likely that the effect of these biases accumulates whenever they are present.”
While evidence-based medicine seeks to be the foundation of clinical practice, it depends on the quality of the evidence and clinician awareness of the nuances within the evidence. The authors note, “up to half of all randomized controlled trials (RCTs) have never been published, and trials with statistically significant findings are more likely to be published than those without,” adding, “negative trials face additional hurdles beyond study publication bias that can result in the disappearance of non-significant results.”
Financial motives for the pharmaceutical industry to restrict the dissemination of unfavorable results is one possibility for skewed representation of negative studies. Still, biases are also present in psychotherapy research. Another explanation for this trend involves the pressures for researchers to produce positive results, as journals with higher visibility publish more positive studies than negative studies.
The authors of this study delineate four major biases:
- Study publication bias: “the non-publication of an entire study.”
- Outcome reporting bias: “the non-publication of negative outcomes within a published article or to switching the status of (non-significant) primary and (significant) secondary outcomes.”
- Spin: “Reporting strategies that could distort the interpretation of results and mislead readers…e.g., instead of concluding that treatment X was not more effective than placebo, concluding that treatment X was well tolerated and was effective in patients who had not received prior therapy.”
- Citation bias: “an obstacle to ensuring that negative findings are discoverable. Studies with positive results receive more citations than negative studies, leading to heightened visibility of positive results.”
To understand the impact of these biases, the researchers assessed the evidence base for antidepressants by identifying 105 depression trials, 74 from a previous study on publication bias and 31 from the Food and Drug Administration (FDA) database, where experiments with non-significant results are accessible, even if unpublished.
Figure 1 illustrates the cumulative impact of study publication bias, outcome reporting bias, spin, and citation bias. 50% (53) of trials were considered positive by the FDA, and 50% (52) were deemed to be negative or questionable. All but one (98%) of the positive studies were published while less than half (48%) of the negative trials were published. In total, 25 were negative of the 77 trials that were published.
10 of the 25 negative trials became positive through either omitting unfavorable outcomes or switching the status of primary and secondary outcomes.
Additionally, 5 of the remaining 15 negative articles were published with a spin in the abstract. For example, “one article reported non-significant results for the primary outcome, yet concluded that the trial ‘demonstrates an antidepressant effect for fluoxetine that is significantly more marked than the effect produced by placebo.’” Similarly
“Only four of 77 published trials unambiguously reported that the treatment was not more effective than placebo in that particular trial,” the study reports. “Compounding the problem, positive trials were cited three times as frequently as negative trials.”
, de Vries and colleagues found that psychotherapy trials “clearly provide a more positive impression of the effectiveness of psychotherapy than is justified by available evidence.” They find that positive psychotherapy trials are cited twice as often as negative trials. However, the researchers found psychotherapy trials more taxing to detect due to the lack of a standardized trial registry. Furthermore, small sample sizes warrant more evidence to make definitive conclusions concerning psychotherapy publication bias.
The researchers offer several approaches to preventing biases:
- Include a regulatory requirement to enforce the mandatory prospective registration for study publication.
- Incorporate behavioral interventions (e.g., psychotherapy) and phase 1 trials (healthy volunteer) within those requirements (until now, they are excluded).
- Closely examine registries by employing independent researchers.
- Publish study protocols, or “registered reports,” in which a journal accepts a study based on the introduction and methods, sans results.
- Encourage peer reviewers to interpret a study’s actual results and check they are accurately reported in abstracts, and that they promote the inclusion of negative study results.
De Vries et al. conclude with practical implications, in which clinicians “must be aware of the potential for bias to distort apparent treatment efficacy, which poses a threat to the practice of evidence-based medicine.”
****
De Vries, Y. A., Roest, A.M., de Jonge, P., Cuijpers, P., Munafò, M.R., Bastiaansen, J.A. (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine 1–3. https://doi.org/10.1017/S0033291718001873 (Link)
An important but neglected topic.
Thank you very much for the review.
So what is the point of “peer review” if this kind of crap is allowed to pass as legit?
Maybe we need a non-peer review by people from a different field!
—- Steve
I think the peers are in on the scam, not to put too fine a point on it. I don’t know who the peers are who review this, but the RCPsych, Kings College and the SMC blatantly authorise false statements (eg the 2 week claim, the “much more effective than placebo” nonsense).
The 2018 Cipriani meta-analysis is a case study in spin. It span the line that AD’s work when their own data showed an effect size of only 0.3 and for the the 2 most popular AD’s, only 0.24 – totally inadequate to justify the “efficacious” label. Taking refuge in rosy looking “response rates” fooled people only until they read the appendices.
What these spin-meisters have achieved is to show it’s all nonsense – the whole psychiatric drug shebang. The cover-up is so crude, so blatant, so dumb frankly, you just know where to look for evidence of deception. If you go ferreting around the anti psychotic studies, the same pattern starts to emerge.
I think psychiatric drug studies have brought medicine into disrepute, aided and abetted by “experts” who are too corrupt and too thick to control it.
It’s been pretty shocking.
Inadvertent bias (lines of research that tend to build on other lines published before) seems to creep in and out of any field, and can narrow people’s focus. But the discovery of findings being deliberately changed and distorted would see the researchers disgraced.
Well, it SHOULD see the researchers disgraced…
There is a pattern I see where the basic study is honestly reported, and then the conclusion is spun. The data shows ADs to be fairly useless, but the conclusion says they are effective, meaning this only in a technical statistical sense. So the papers conclusion is highly tenous and nearly a flat out lie but not quite. It’s as if you have to conclude ADs work despite your data.
Then the press release and surrounding statements take that conclusion and amplify it to dishonest levels.
The profession actively wants dishonest reporting to justify the status quo, they are desperately covering up the increasingly obvious catastrophy. I may be an optimist, but I do think the genie is out of the bottle and the game is up, as carers, journos, insiders and patients are quick to voice their disbelief.
It’s a massive conflict of interest situation. They’re not going to honestly report things that will hurt their bottom lines. So it’s more marketing than it is science, but sadly, most people don’t know or want to know this.
Problem is, so few people read (or understand) the appendices. I’d hazard a guess that most folks only ever read the Abstract and Conclusions sections, and take that as truth. But so often, the statements made in these sections actively contradict the actual data! We really lost a gem when Mickey Nardo died. He was so very good at scouring the research for this crap and bringing the contradictions/lies to light.
“I think psychiatric drug studies have brought medicine into disrepute, aided and abetted by ‘experts’ who are too corrupt and too thick to control it.” I agree.
The systemic child abuse cover uppers of both the psychological and psychiatric industries need to be judged fairly for whom they have chosen to become.