There has been a lot of attention on clinical trial registration over the last decade. Essentially, because of some very clever and courageous researchers and clinicians, including Peter Gøtzsche, Ben Goldacre, Irving Kirsch and many many others, the public have increasingly become aware that the literature base to support medications, including psychiatric medications, is tainted and biased at best and fraudulent at worst.
What has been exposed is that medications, that we have been led to believe are evidenced based, are not as good as we thought they were. Negative trials haven’t been published, and researchers have been changing primary outcome measures (POMs; the measures they should have decided on before collecting data) and the time frames in which they intended to look at whether a drug was effective or not. For some medications, the difference between placebo and drug effect is small and often not clinically significant.
To address the bias occurring in the medical literature associated with selective outcome reporting, in 2005, the International Committee of Medical Journal Editors (ICMJE; www.icmje.org) introduced mandatory trial registration guidelines and member journals required prospective registration of trials prior to patient enrolment as a condition of publication. To date, no research has examined whether these guidelines are impacting psychiatry publications.
In February 2013, soon after Ben Goldacre published his bestseller, Bad Pharma, an honours student, Amelia Scott, walked in to my office looking for a project. Inspired by some of Ben’s ideas, we decided to ask a simple question. Do journals stipulating that as a condition of publication a clinical trial must be prospectively registered actually do what they say they do? Now this may not seem like a very important question, but scientists who publish know that there are rules for submission and the assumption is that those rules are followed. Why else would a journal say that a trial must be prospectively registered and then not require it? If this doesn’t happen, then how far have we really come in cleaning up the literature base and how much faith can we have in studies that are currently being published? But if we did find that journals were following the standard they set, then fantastic. We would show that the system was working and that everyone was working to meet these new standards.
So What Did We Do?
We investigated the top 5 journals that mandated prospective registration as part of the conditions of publication. These were the American Journal of Psychiatry, Archives of General Psychiatry/JAMA Psychiatry, Biological Psychiatry, Journal of the American Academy of Child and Adolescent Psychiatry, and the Journal of Clinical Psychiatry. We based the definition of “top” journal on impact factor which, while not perfect, is a reasonable way to establish influence and reputation.
We identified all clinical trials (as defined by ICMJE) that had been conducted after July 2005 (or 2007 for two journals) and published between January 2009 and July 2013. For each identified trial, where possible, we extracted trial registration information, changes to POMs between publication and registry to assess selective outcome reporting, changes to participant numbers, and funding type.
And What Did We Find?
Out of 3305 articles, 181 studies were identified as clinical trials requiring registration (that is, they should all have been prospectively registered because that’s what the journals that published the trial require).
Twenty-one (11.6%) of the 181 studies were deemed unregistered, 61 (33.7%) were retrospectively registered, 37 (20.4%) had unclear POMs either in the article or the registry and 2 (1.1%) were registered in an inaccessible trial registry. Only 60 (33.1%) studies were prospectively registered with clearly defined POMs.
BUT seventeen of these 60 (28.3%) properly registered trials showed evidence of selective outcome reporting – this means that there had been changes to POMs based on a comparison of the trial registry and the publication. In total, only 26 (14.4%) of the 181 the trials were prospectively registered and did not alter their POMs or the time frames at which they were measured. Prospective registration with no changes in POMs occurred more frequently with pharmaceutical funding.
Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs. Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.
And How Were These Results Received by the Journals We Investigated?
We submitted this work to two of the five journals we investigated. One journal (JAMA Psychiatry) refused to review it, stating the paper was too focused for their journal, and the American Journal of Psychiatry reviewed it, rejected it, accepted an appeal, allowed it to be re-reviewed and then rejected it again. Clearly AJP didn’t like it. We also tried JAMA (rejected without review), BMJ (rejected without review), the Lancet (rejected without review –they at least said it was interesting but not interesting enough), and Journal of the American Medical Informatics Association (rejected without review). We finally found a home with PLoS One, where it has just been published. All the data have been submitted if someone wants to verify our findings.
So What Does This Mean?
Well it might mean nothing at all. It is possible that all those researchers who didn’t preregister their trials or who changed their outcome measures before publication did so for very benign reasons and that we can trust their results on their word without having a registry to check up on whether they did what they said they were going to do.
Should We Believe This?
History suggests that when left unchecked, researchers have been known to change their data. As demonstrated by case studies on paroxetine (1), burying primary outcome data because it is not statistically significant can have dangerous implications for the individuals who eventually end up using the treatment. Changing time frames can result in a drug appearing more effective as compared with placebo, as evidenced by the reporting of Ballenger et al (2) on Alprazolam. In this study, the researchers focused on the 4 week data despite the fact that the trial was 8 weeks. While at 4 weeks there was clear advantage of the drug over placebo, this advantage disappeared at the 8 week time point. People glorify their positive results and minimize or neglect reporting on negative results. After all that’s why the registry was developed in the first place: to change behaviour.
At worst, our findings mean that the trials published over the last decade cannot be fully trusted. And given that health decisions and funding are based on these published findings, we should be very concerned.
What Can be Done to Improve the Situation?
Looking forward, the situation could be improved if some of the following suggestions were to be adopted:
1) Member journals of the ICMJE should have a dedicated person checking trial registries, trials should simply not be published if they haven’t been prospectively registered as determined by the ICMJE or the journals should state clearly and transparently reasons why studies might be published without adhering to ICMJE guidelines.
2) If authors do change POMs or participant numbers or retrospectively register their trials, the reasons should be clearly outlined in the methods section of the publication.
3) To further improve transparency, authors could upload the full clinical trial protocol, including all amendments, to the registry website and provide the raw data from a clinical trial in a format accessible to the research community.
4) Greater effort needs to be made to ensure authors are aware of the importance of prospectively registering trials, by improving guidelines for submission (3) and when applying for ethical approval.
5) Finally, reviewers should not make decisions about the acceptability of a study for publication based on whether the findings are positive or negative as this may be implicitly encouraging authors to be selective in reporting results.
Indeed, a recent study showed that only one third of peer reviewers examined registered trial information and reported any discrepancies to the journal editors (4). It is clear that while there are guidelines in place to decontaminate the medical literature, authors, reviewers and journal editors need to work together to ensure that these guidelines are upheld.
What’s next?
We are investigating whether psychology is better behaved than psychiatry.
To read our entire article, it is freely available at PloS One:
Scott A, Rucklidge JJ, Mulder RT. Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PLoS One 10:e0133718, 2015.
* * * * *
References:
1. Zarin DA, Tse T, Ide NC. Trial Registration at ClinicalTrials.gov between May and October 2005. N Engl J Med. 2005;353:2779-2787.
2. Ballenger JC, Burrows GD, DuPont RL, Jr, et al. Alprazolam in panic disorder and agoraphobia: Results from a multicenter trial: i. efficacy in short-term treatment. Arch Gen Psychiatry. 1988;45:413-422.
3. Knuppel H, Metz C, Meerpohl JJ, Strech D. How psychiatry journals support the unbiased translation of clinical research. A cross-sectional study of editorial policies. PLoS One. 2013;8:e75995.
4. Mathieu S, Chan AW, Ravaud P. Use of trial register information during the peer review process. PLoS One. 2013;8:e59910.
Thank you for this interesting article. A new study seems to answer the question why preregistration is frowned upon:
“The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000.”
http://www.nature.com/news/registered-clinical-trials-make-positive-findings-vanish-1.18181?WT.mc_id=TWT_NatureNews
Report comment
Thanks for this article. Given that corporations function as sociopathic amoral opportunistic profit-making machines, it’s a safe bet that institutional researchers tied to Big Pharma are manipulating the data to the maximum extent allowed by the (lack of enforced) rules.
It is hard to put into words the disdain and disgust I feel toward these type of psychiatric “studies”. No matter how much reform is done, these studies will continue to rely on unreliable invalid labels like depression or schizophrenia. Therefore, the results from such studies will be suspect regardless of how they are registered.
But perhaps reforming the registration process will make it harder to manufacture the perception of “value” for psychiatric drugs. If so it will be another push bringing the myth of psychiatric drugs’ effectively treating life problems closer to collapse.
Report comment
Hi Dr. Ruckledge,
I was interested in this article as I used to work at a database company that tracks the progress of drugs in clinical trials, in part by using the clinical trial registry updates. I was surprised by your findings, as it conflicts with my personal experience working with the subset of trials that are conducted for purposes of regulatory submissions of new drugs or new indications for approved drugs.
Starting at the end of your excel spreadsheet of retrospectively registered trials and working upwards, one finds a phase 3 registrational trial of the antipsychotic drug ziprasidone, registered in January 2006 but beginning in December 2005. In a strictly legalistic sense, this is indeed a “retrospectively registered trial”, but the implication that this retrospective registration somehow allowed for retrospectively changing the endpoints seems unsupported.
* This was a registrational trial conducted for regulatory approval in the United States, and so the trial design would have been disclosed to FDA prior to beginning enrollment
* It was a double blind trial, with the blind unbroken until the primary completion date of May 2008. With the allocation of subjects to the treatment arm blinded, one cannot “adjust” the endpoints to optimize the apparent effiacy
*As the endpoints of the trial were measured after 8 weeks of treatment, no final data for any patient would be available at the time the trial was registered, even if the blind were broken.
* Large phase 3 trials typically take from 9 months to 1.5 years to recruit the full number of subjects. By the end of January 2006, it is unlikely that more than 50 of the 504 subjects in this trial had even been enrolled.
Working up the table, one finds similarly that the trials in rows 61, 60, 59, 58, 56, 54, 53, and 52 were double-blind trials for which results would not have been known to the investigators at the time the trial was registered (1-2 months after the start date). Those in rows 57, 55, 51, and 50 were either clearly registered after results were available or the timing at which results became available was ambiguous. I did not examine the other 48 trials in detail.
What perecentage of these trials overall were double blinded, with the trial registered prior to the Primary Completion date? I would think that this would be a better measure of the potential for retrospective adjustment of the endpoints.
Thanks for making the raw data available in such a convenient form by the way.
Report comment
Thank you for your comments and spending the time looking at our raw data! We completely agree with you on the points you raise retiming of registration and indeed, in the article, we divided up retrospective registration into different time frames in Table 1 for the very reason that we thought if a research group had registered a month after the trial started the likelihood of changing POMs as a result of seeing results was highly unlikely, as you state. And indeed there were 7 trials registered within one month and 14 within 1 to 6 months. However, almost half of the retrospectively registered trials were registered between 1 and 5 years after patient recruitment had started. At this point could they be influenced by data already collected? We would hope not if the trials were fully blinded although we know that patients often can figure out what group they are in based on side effect profile. Does that influence investigators? We don’t know.
We are completely sympathetic to a late registration within months of starting a study. This has happened in my lab in the days of not knowing about trial registration and during the disruption of the Canterbury earthquakes when our department was closed for 3 months. There can be benign reasons for it! However, I would then never have thought to submit this work to a journal stipulating prospective trial registration as a condition of publication!
What we would simply ask for is that the journals be transparent about their procedures. Given the bleak history of trials in psychiatry, it is important that journals take measures to ensure that we can trust what is currently published and if they deviate, they simply let the reader know by ensuring it is stated in the article.
Amelia developed amazing detective skills as part of this study. We were surprised at the changes that were happening in the registry for prospectively registered trials. Some were benign changes, others were less so. We didn’t look at this for those that were retrospectively registered. Perhaps we should have!
A good question about whether they were double blind or not. Trials did not have to be double blind to be included although many were. That would have been a good point to make in our discussion. Thanks for that observation.
Report comment
One more interesting thing would be to see if the trials which were unregistered and/or changed tehir procedures after registration had a higher rate of positive findings than the properly registered and conducted studies.
Report comment
Thank you for your thoughts and response Dr. Rucklidge.
This is certainly an interesting study. It might be nice to follow it up with some additional analysis on time trends, breakdown by intervention type (there were both psychotherapy and drug trials in the dataset), and trial phase (or size).
Report comment
Yes we thought about doing that but because of the lag in publication we didn’t think we would get an accurate look across time. We are currently looking at psychotherapy trials only. So watch this space.
Report comment
An excellent study that was obviously a great deal of work. Thanks for doing it. I think your worksheets with the specific articles and their foibles might be useful to others vetting these distorted studies if posted publicly…
Report comment
I recently discovered clinical studies done on children for children and adolescents: This is taken exactly from the text. Celexa – no therapeutic effect between citalipram and placebo. Lamictal – pharmacokenetics for Lamictal on babies (1-24 months) This should be illegal! Provigal – 6-16 yrs, study failed, nothing was concluded. Risperdal for Irritability Associated with Autism Disorder – There are no controled data to support longer term treatment of Risperdal beyond 8 weeks in adolescents with schizophrenia. Zoloft 6-11 years – The following 12 pages that provided data concerning safety and efficacy have been deleted from view. All conclusions for this study have also been deleted from view. Meridia classified as an SNRI did show in this pediatric study 2 suicide attempts, suicide ideation, depression, accidental injury, and increased adverse Events. All conclusions and summaries have been deleted from view. Effexor 7-17 year olds – Executive Summary: Recomended ‘non-approvable’, failure to show efficacy over placebo for MDD and GAD. and finally, 2 clinical trials for Abilify 6-17 year old for Irritability Associated with Autistic Disorder – Conclusion and Summary: both studies FAILED. And we all know about Paxil 329. Then I researched just how many children and adolescents were being prescribed these drugs and the numbers were so astronomical I could not wrap my head around it. This to me is travesty, injustice, and cruel and unusual punishment society is putting our children through. Just unbelievable…..
Report comment