Is Mandatory Trial Registration Decontaminating the Psychiatric Literature?

Julia Rucklidge, PhD
9
99

There has been a lot of attention on clinical trial registration over the last decade. Essentially, because of some very clever and courageous researchers and clinicians, including Peter Gøtzsche, Ben Goldacre, Irving Kirsch and many many others, the public have increasingly become aware that the literature base to support medications, including psychiatric medications, is tainted and biased at best and fraudulent at worst.

What has been exposed is that medications, that we have been led to believe are evidenced based, are not as good as we thought they were. Negative trials haven’t been published, and researchers have been changing primary outcome measures (POMs; the measures they should have decided on before collecting data) and the time frames in which they intended to look at whether a drug was effective or not. For some medications, the difference between placebo and drug effect is small and often not clinically significant.

To address the bias occurring in the medical literature associated with selective outcome reporting, in 2005, the International Committee of Medical Journal Editors (ICMJE; www.icmje.org) introduced mandatory trial registration guidelines and member journals required prospective registration of trials prior to patient enrolment as a condition of publication. To date, no research has examined whether these guidelines are impacting psychiatry publications.

In February 2013, soon after Ben Goldacre published his bestseller, Bad Pharma, an honours student, Amelia Scott, walked in to my office looking for a project. Inspired by some of Ben’s ideas, we decided to ask a simple question. Do journals stipulating that as a condition of publication a clinical trial must be prospectively registered actually do what they say they do? Now this may not seem like a very important question, but scientists who publish know that there are rules for submission and the assumption is that those rules are followed. Why else would a journal say that a trial must be prospectively registered and then not require it? If this doesn’t happen, then how far have we really come in cleaning up the literature base and how much faith can we have in studies that are currently being published? But if we did find that journals were following the standard they set, then fantastic. We would show that the system was working and that everyone was working to meet these new standards.

So What Did We Do?

We investigated the top 5 journals that mandated prospective registration as part of the conditions of publication. These were the American Journal of Psychiatry, Archives of General Psychiatry/JAMA Psychiatry, Biological Psychiatry, Journal of the American Academy of Child and Adolescent Psychiatry, and the Journal of Clinical Psychiatry. We based the definition of “top” journal on impact factor which, while not perfect, is a reasonable way to establish influence and reputation.

We identified all clinical trials (as defined by ICMJE) that had been conducted after July 2005 (or 2007 for two journals) and published between January 2009 and July 2013. For each identified trial, where possible, we extracted trial registration information, changes to POMs between publication and registry to assess selective outcome reporting, changes to participant numbers, and funding type.

And What Did We Find?

Out of 3305 articles, 181 studies were identified as clinical trials requiring registration (that is, they should all have been prospectively registered because that’s what the journals that published the trial require).

Twenty-one (11.6%) of the 181 studies were deemed unregistered, 61 (33.7%) were retrospectively registered, 37 (20.4%) had unclear POMs either in the article or the registry and 2 (1.1%) were registered in an inaccessible trial registry. Only 60 (33.1%) studies were prospectively registered with clearly defined POMs.

BUT seventeen of these 60 (28.3%) properly registered trials showed evidence of selective outcome reporting – this means that there had been changes to POMs based on a comparison of the trial registry and the publication. In total, only 26 (14.4%) of the 181 the trials were prospectively registered and did not alter their POMs or the time frames at which they were measured. Prospective registration with no changes in POMs occurred more frequently with pharmaceutical funding.

Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs. Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.

And How Were These Results Received by the Journals We Investigated?

We submitted this work to two of the five journals we investigated. One journal (JAMA Psychiatry) refused to review it, stating the paper was too focused for their journal, and the American Journal of Psychiatry reviewed it, rejected it, accepted an appeal, allowed it to be re-reviewed and then rejected it again. Clearly AJP didn’t like it. We also tried JAMA (rejected without review), BMJ (rejected without review), the Lancet (rejected without review –they at least said it was interesting but not interesting enough), and Journal of the American Medical Informatics Association (rejected without review). We finally found a home with PLoS One, where it has just been published.  All the data have been submitted if someone wants to verify our findings.

So What Does This Mean?

Well it might mean nothing at all. It is possible that all those researchers who didn’t preregister their trials or who changed their outcome measures before publication did so for very benign reasons and that we can trust their results on their word without having a registry to check up on whether they did what they said they were going to do.

Should We Believe This?

History suggests that when left unchecked, researchers have been known to change their data. As demonstrated by case studies on paroxetine (1), burying primary outcome data because it is not statistically significant can have dangerous implications for the individuals who eventually end up using the treatment. Changing time frames can result in a drug appearing more effective as compared with placebo, as evidenced by the reporting of Ballenger et al (2) on Alprazolam. In this study, the researchers focused on the 4 week data despite the fact that the trial was 8 weeks. While at 4 weeks there was clear advantage of the drug over placebo, this advantage disappeared at the 8 week time point. People glorify their positive results and minimize or neglect reporting on negative results. After all that’s why the registry was developed in the first place: to change behaviour.

At worst, our findings mean that the trials published over the last decade cannot be fully trusted. And given that health decisions and funding are based on these published findings, we should be very concerned.

What Can be Done to Improve the Situation?

Looking forward, the situation could be improved if some of the following suggestions were to be adopted:

1) Member journals of the ICMJE should have a dedicated person checking trial registries, trials should simply not be published if they haven’t been prospectively registered as determined by the ICMJE or the journals should state clearly and transparently reasons why studies might be published without adhering to ICMJE guidelines.

2) If authors do change POMs or participant numbers or retrospectively register their trials, the reasons should be clearly outlined in the methods section of the publication.

3) To further improve transparency, authors could upload the full clinical trial protocol, including all amendments, to the registry website and provide the raw data from a clinical trial in a format accessible to the research community.

4) Greater effort needs to be made to ensure authors are aware of the importance of prospectively registering trials, by improving guidelines for submission (3) and when applying for ethical approval.

5) Finally, reviewers should not make decisions about the acceptability of a study for publication based on whether the findings are positive or negative as this may be implicitly encouraging authors to be selective in reporting results.

Indeed, a recent study showed that only one third of peer reviewers examined registered trial information and reported any discrepancies to the journal editors (4). It is clear that while there are guidelines in place to decontaminate the medical literature, authors, reviewers and journal editors need to work together to ensure that these guidelines are upheld.

What’s next?

We are investigating whether psychology is better behaved than psychiatry.

To read our entire article, it is freely available at PloS One:

Scott A, Rucklidge JJ, Mulder RT. Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PLoS One 10:e0133718, 2015.

* * * * *

References:

1.  Zarin DA, Tse T, Ide NC. Trial Registration at ClinicalTrials.gov between May and October 2005. N Engl J Med. 2005;353:2779-2787.

2.  Ballenger JC, Burrows GD, DuPont RL, Jr, et al. Alprazolam in panic disorder and agoraphobia: Results from a multicenter trial: i. efficacy in short-term treatment. Arch Gen Psychiatry. 1988;45:413-422.

3.  Knuppel H, Metz C, Meerpohl JJ, Strech D. How psychiatry journals support the unbiased translation of clinical research. A cross-sectional study of editorial policies. PLoS One. 2013;8:e75995.

4.  Mathieu S, Chan AW, Ravaud P. Use of trial register information during the peer review process. PLoS One. 2013;8:e59910.

Previous articleFDA Approves Controversial “Pink Viagra”
Next articleDelusions
Julia Rucklidge, PhD
Nutrition and Mental Health: Julia's interest in nutrition and mental illness grew out of her own research showing poor outcomes for children with psychiatric illness despite conventional treatments. She has been investigating the role of micronutrients in mental illness.

Support MIA

Enjoyed what you just read? Consider a donation to help us continue to produce content, provide up-to-date research news, offer continuing education courses, and continue building a community for exploring alternatives to the current paradigm of mental health. All donations are tax deductible.

$
Select Payment Method
Loading...
Personal Info

Credit Card Info
This is a secure SSL encrypted payment.

Donation Total: $20.00

9 COMMENTS

  1. Thank you for this interesting article. A new study seems to answer the question why preregistration is frowned upon:

    “The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000.”
    http://www.nature.com/news/registered-clinical-trials-make-positive-findings-vanish-1.18181?WT.mc_id=TWT_NatureNews

  2. Thanks for this article. Given that corporations function as sociopathic amoral opportunistic profit-making machines, it’s a safe bet that institutional researchers tied to Big Pharma are manipulating the data to the maximum extent allowed by the (lack of enforced) rules.

    It is hard to put into words the disdain and disgust I feel toward these type of psychiatric “studies”. No matter how much reform is done, these studies will continue to rely on unreliable invalid labels like depression or schizophrenia. Therefore, the results from such studies will be suspect regardless of how they are registered.

    But perhaps reforming the registration process will make it harder to manufacture the perception of “value” for psychiatric drugs. If so it will be another push bringing the myth of psychiatric drugs’ effectively treating life problems closer to collapse.

  3. Hi Dr. Ruckledge,

    I was interested in this article as I used to work at a database company that tracks the progress of drugs in clinical trials, in part by using the clinical trial registry updates. I was surprised by your findings, as it conflicts with my personal experience working with the subset of trials that are conducted for purposes of regulatory submissions of new drugs or new indications for approved drugs.

    Starting at the end of your excel spreadsheet of retrospectively registered trials and working upwards, one finds a phase 3 registrational trial of the antipsychotic drug ziprasidone, registered in January 2006 but beginning in December 2005. In a strictly legalistic sense, this is indeed a “retrospectively registered trial”, but the implication that this retrospective registration somehow allowed for retrospectively changing the endpoints seems unsupported.

    * This was a registrational trial conducted for regulatory approval in the United States, and so the trial design would have been disclosed to FDA prior to beginning enrollment
    * It was a double blind trial, with the blind unbroken until the primary completion date of May 2008. With the allocation of subjects to the treatment arm blinded, one cannot “adjust” the endpoints to optimize the apparent effiacy
    *As the endpoints of the trial were measured after 8 weeks of treatment, no final data for any patient would be available at the time the trial was registered, even if the blind were broken.
    * Large phase 3 trials typically take from 9 months to 1.5 years to recruit the full number of subjects. By the end of January 2006, it is unlikely that more than 50 of the 504 subjects in this trial had even been enrolled.

    Working up the table, one finds similarly that the trials in rows 61, 60, 59, 58, 56, 54, 53, and 52 were double-blind trials for which results would not have been known to the investigators at the time the trial was registered (1-2 months after the start date). Those in rows 57, 55, 51, and 50 were either clearly registered after results were available or the timing at which results became available was ambiguous. I did not examine the other 48 trials in detail.

    What perecentage of these trials overall were double blinded, with the trial registered prior to the Primary Completion date? I would think that this would be a better measure of the potential for retrospective adjustment of the endpoints.

    Thanks for making the raw data available in such a convenient form by the way.

  4. Thank you for your thoughts and response Dr. Rucklidge.

    This is certainly an interesting study. It might be nice to follow it up with some additional analysis on time trends, breakdown by intervention type (there were both psychotherapy and drug trials in the dataset), and trial phase (or size).

  5. I recently discovered clinical studies done on children for children and adolescents: This is taken exactly from the text. Celexa – no therapeutic effect between citalipram and placebo. Lamictal – pharmacokenetics for Lamictal on babies (1-24 months) This should be illegal! Provigal – 6-16 yrs, study failed, nothing was concluded. Risperdal for Irritability Associated with Autism Disorder – There are no controled data to support longer term treatment of Risperdal beyond 8 weeks in adolescents with schizophrenia. Zoloft 6-11 years – The following 12 pages that provided data concerning safety and efficacy have been deleted from view. All conclusions for this study have also been deleted from view. Meridia classified as an SNRI did show in this pediatric study 2 suicide attempts, suicide ideation, depression, accidental injury, and increased adverse Events. All conclusions and summaries have been deleted from view. Effexor 7-17 year olds – Executive Summary: Recomended ‘non-approvable’, failure to show efficacy over placebo for MDD and GAD. and finally, 2 clinical trials for Abilify 6-17 year old for Irritability Associated with Autistic Disorder – Conclusion and Summary: both studies FAILED. And we all know about Paxil 329. Then I researched just how many children and adolescents were being prescribed these drugs and the numbers were so astronomical I could not wrap my head around it. This to me is travesty, injustice, and cruel and unusual punishment society is putting our children through. Just unbelievable…..