A new study, published in Psychological Medicine, found evidence for a specific type of publication bias distorting the evidence about antidepressant efficacy. Negative studies (studies that found that antidepressants were ineffective) were far more likely to be published only in pooled-trials studies, usually addressing secondary questions. On the other hand, positive studies were likely to be published as stand-alone publications touting the effectiveness of antidepressants, in addition to pooled-trials studies. This skews the published evidence for the primary question of antidepressant efficacy.
Pooling data is when researchers combine the results from several trials into one analysis. For instance, a company may conduct nine experiments of its medication, which may be pooled together into one report. This way, even if, say, four of the trials are negative, it may still appear to be a useful product. Additionally, pooled-trials studies often look at different questions. That is, instead of asking “is this antidepressant effective,” a pooled-trials study may ask about other outcome measures, adverse events, or may address questions about dosage or particular groups for whom the medication might be more effective. While this can be useful information, it cannot replace the initial question of whether the antidepressant outperforms placebo.
“Pooled-trials publications flood the evidence base with often-redundant articles that, instead of addressing the original primary research question, present (positive) results on secondary questions,” according to the researchers. “Therefore, pooled-trials publications distort the apparent risk-benefit profile of antidepressants.”
Ymkje Anna de Vries led the research at the University of Groningen, The Netherlands. The researchers conducted a systematic search for pooled-trials publications addressing antidepressant efficacy for 16 well-known antidepressants. They found 107 such publications—only 13 of which addressed the primary question (are antidepressants efficacious). Only 4 of those publications included individual-level data on that question.
De Vries and the other researchers found that 96.2% of the positive trials identified in pooled publications were also published as stand-alone evidence for antidepressant efficacy. However, only about half (58.9%) of the negative trials were also published elsewhere. All of the positive trials of antidepressants were published in one form or another, while 17.6% of the negative trials were not even published in pooled-trials publications.
Interestingly, of the 105 separate trials of antidepressant medications that were included in the study, 54 trials were rated as “effective,” and 51 trials were rated “ineffective” by the FDA—which means that a pooled study might conclude that the evidence for antidepressant efficacy could be compared to a coin toss. However, only 5 of the 107 pooled-trials publications had a negative conclusion.
Publication bias, which in general means that positive trials are likely to be published, while negative trials are not, has been well-documented in the medical field. It can occur for a number of reasons, including journal editors rejecting negative trials because they appear unimportant, or because it is in the interests of the sponsor of the trial (such as the pharmaceutical industry) that their product appears effective (i.e. “hiding” negative results).
The researchers note that 91% of the pooled-trials publications had at least one member of the research team employed by the pharmaceutical industry, raising questions of conflict-of-interest.
Pooled-trial publications add to this bias for several reasons, according to de Vries and the other researchers. For instance, they may not address the primary question of efficacy at all; trials may be selected to be included in a biased manner, or the results from positive trials may be presented in such a way as to overshadow the concerns that negative studies may raise.
Additionally, the pharmaceutical industry has argued that pooled-trial publications count towards publishing all their results—so it could be argued that they do not need to publish the results of negative trials again. This is particularly problematic because the positive trials are being published again, in most cases.
The researchers also noticed that there were redundant publications for certain medications, especially for duloxetine, venlafaxine, and escitalopram. That is, one trial’s results were repeatedly published for these antidepressants, giving the impression that there was more evidence for their efficacy.
“The overall effect of these pooled trials publications is to flood the evidence base with articles that encourage the use of a drug, while discouraging results from the original trials remain invisible.”
De Vries, Y.A., Roest, A.M., Turner, E.H., de Jonge, P. (2018). Hiding negative trials by pooling them: a secondary analysis of pooled-trials publication bias in FDA registered antidepressant trials. Psychological Medicine, 1–7. https://doi.org/10.1017/ S0033291718002805 (Link)