A new study suggests that the effectiveness of antipsychotic medications may be overstated due to biased publication methods. The researchers found that the vast majority of antipsychotic drug trials used biased reporting methods. These practices ensured that the published drug trials demonstrated positive outcomes by obscuring a number of negative findings.
Since 2006, when the World Health Organization mandated it, drug trials are required to pre-register their intended outcome measures on the clinicaltrials.gov database. This was designed to prevent selective reporting bias, a type of deceptive practice in which researchers conduct many outcome measures, then report only the ones with positive results.
Researchers at Utrecht University conducted the current study. The corresponding author was Jurjen J. Luykx at the Department of Translational Neuroscience. Luykx and his colleagues found that 85% of the published antipsychotic drug trials did not report on the outcome measures they indicated when they pre-registered their study. In fact, according to the researchers:
“81% of [antipsychotic drug trials] had at least one secondary outcome non-reported, newly introduced, or changed to a primary outcome in the respective publication.”
When researchers conduct studies to determine whether medications are effective, it is standard practice to declare a primary outcome (such as scores on a standardized measure of psychosis). However, when researchers use multiple questionnaires assessing many kinds of outcomes, it increases the chances that any one outcome may be improved simply due to chance. That is, the more tests you run, the more likely it is that one of them will find a significant result.
When researchers report on the test that came out positive, rather than the test they selected as their primary outcome at the beginning of the study, this is known as selective outcome reporting bias. This allows researchers to ensure that their results will be positive and publishable, but has the side effect of deceiving the public—and potentially other researchers—into thinking that the results were uniformly positive.
Luykx and his colleagues found 48 published randomized control trials of antipsychotic medications for schizophrenia or schizophreniform disorder since 2006 and compared their reported outcomes with the outcome measures they had registered in the clinical trials database before beginning their studies. However, 17 of the 48 trials did not register in the database until after their research was complete, defeating the purpose of pre-registering outcomes.
Of the 48 trials, four did not even mention their pre-registered primary outcome in their publication. Another three converted their pre-registered primary outcome to a secondary outcome when publishing (that is, making the primary outcome seem less important). A further ten studies failed to appropriately pre-specify their primary outcome.
Secondary outcomes showed even more discrepancies. Eighteen of the trials failed to mention their pre-specified secondary outcomes in their publication; four of the studies converted a secondary outcome to a primary outcome for publication (making it seem as though it was the target outcome, when in fact it was a minor part of the study); and 37 of the 48 trials failed to pre-specify all of their secondary outcomes.
Regarding the safety and tolerability of antipsychotic drugs, there were 74 pre-specified outcomes—of which 53 were included in the relevant publications. However, the papers added 335 new outcome measures regarding safety and tolerability that were not pre-registered. This may indicate that researchers conducted a myriad of safety and tolerability tests to selectively publish the ones that were favorable.
These findings are consistent with previous research that found similar results for both antipsychotic drug trials (see full article) and the antidepressant drug trial literature (see full articles here and here). These studies document other forms of bias as well, such as the tendency to leave negative results unpublished. For instance, in antidepressant trials, when published and unpublished trials are combined, 49% of the trials found that the drugs were no more effective than placebo.
Luykx and his fellow researchers suggest that the research community needs to be more careful. Authors must explain discrepancies in their published articles, and journal editors must demand that researchers hold to their pre-specified outcome measures.
Only when these pervasive sources of bias are reduced will the public be able to trust the reliability of the studies testing the efficacy and safety of these drugs.
Lancee, M., Lemmens, C. M. C., Kahn, R. S., Vinkers, C. H., & Luykx, J. J. (2017). Outcome reporting bias in randomized-controlled trials investigating antipsychotic drugs. Translational Psychiatry, 7, e1232. doi:10.1038/tp.2017.203 (Link)