The Cochrane Collaboration was once one of the most trusted institutions in healthcare. People trusted Cochrane systematic reviews, even those of psychiatric drugs. However, in 2018, a highly skilled researcher in Cochrane, Professor Tom Jefferson, said in an interview in the article, “Cochrane – a sinking ship?”: “If your review is made up of studies which are biased and in some cases are ghost written or the studies are cherry picked and you don’t take that into account in your review, then it’s garbage in and garbage out … with a nice little Cochrane logo on it.”
About the failing scientific integrity of Cochrane reviews, Cochrane’s major funder announced in 2019 that, “This is a point raised by people in the Collaboration to ensure that garbage does not go into the reviews; otherwise, your reviews will be garbage.” Four months later, the funder announced that their funding of Cochrane review groups based in the UK would cease at the end of March 2023. I have described these issues in my freely available book, The Decline and Fall of the Cochrane Empire.
Today, most of the UK based Cochrane review groups have closed. Many see this as the beginning to the end for a Cochrane that cared too little about scientific rigour and too much about protecting guild interests and other financial interests, and which had become too close to industry.
As I have demonstrated in my books and articles, almost all Cochrane reviews of psychiatric drugs are garbage-in, garbage-out. This, I confirmed in the newest Cochrane meta-analysis of depression pills in children, for which the first author is Sarah Hetrick, co-ordinating editor in the Cochrane Mental Disorders group.
I shall give another recent example that Cochrane is not an open-minded institution that tries to ferret out the truth, but one that defends psychiatrists’ common beliefs, regardless of how idiotic they may be.
When people are acutely disturbed with psychotic symptoms, short-term drugs can sometimes be of benefit, as the patients might need to be calmed down and get some sleep. But which drugs? Benzodiazepines or neuroleptics? It is routine to use neuroleptics but is this preferable?
In a 2017 Cochrane review about benzodiazepines for psychosis-induced aggression or agitation, the authors wrote in the abstract that there was no observed effect for benzodiazepines when compared with haloperidol. This was seriously misleading. The confidence interval included the possibility that benzodiazepines are similarly effective as haloperidol, and one cannot say there is no effect of benzodiazepines without having a placebo control in the studies.
In June 2018, I wrote to the first author of the review, Hadar Zaman, and asked him to correct the misleading abstract. He forwarded my comments to the Cochrane Schizophrenia Group and said they would come back with guidance.
They didn’t. Three months later, I wrote to Zaman again, copying the Managing Editor of the Cochrane Schizophrenia Group, Claire Irving:
“I have patiently waited for 3 months for a reply but have not been contacted by the review group. I have therefore sent my criticism today via the Comments function in the [Cochrane] Library. I copy the managing editor. Apart from this, you could easily yourself have made the small corrections to the review I asked for, without involving the editor before you had made them.”
A unique feature of Cochrane reviews is that comments and criticism are invited, which are then published with the review, and the review itself can be improved.
However, yet again, Cochrane ignored my comment. Irving replied that the group would respond “as soon as possible.”
Three years later, I had still not heard from the group.
I therefore submitted my comment to the group again, on 25 August 2021:
Misleading denigration of benzodiazepines when compared to antipsychotics
In this review about benzodiazepines for psychosis-induced aggression or agitation, the authors write in the abstract:
“When compared with haloperidol, there was no observed effect for benzodiazepines for sedation by 16 hours (n = 434, 8 RCTs, RR 1.13, 95% CI 0.83 to 1.54, low quality evidence).”
I find this misleading. Assuming haloperidol works, it seems that benzodiazepines also work. One cannot say that benzodiazepines have no effect unless the comparator is placebo, and in fact, we already know that benzodiazepines can calm people down. They have been used as rescue medication in many trials of depression pills when the patients became agitated because of the pills (1). The abstract therefore needs to be changed.
I also wonder why the authors did not quote a similar Cochrane review: “Dold M, Li C, Tardy M, et al. Benzodiazepines for schizophrenia. Cochrane Database Syst Rev 2012;11:CD006391.”
The authors of this review wrote: “There is currently no convincing evidence to confirm or refute the practice of administering benzodiazepines as monotherapy.” I commented on this review in my book about psychiatry (1):
“In actual fact, we should use benzodiazepines instead of antipsychotics. In 14 trials that had compared them, the desired sedation occurred significantly more often on benzodiazepines. Benzodiazepines were compared with placebo in eight trials, and the authors reported that the proportion of treatment failures wasn’t significantly lower with benzodiazepines than with placebo (six trials, 382 patients, relative risk 0.67, 95% CI 0.44 to 1.02). My interpretation of these data is entirely different. Of course benzodiazepines calm patients down, which a relative risk of 0.67 also implies. Whether or not it is statistically significant doesn’t matter; it would have become significant with a few more patients. So why don’t psychiatrists use benzodiazepines instead of antipsychotics? And if they find the trials of poor quality, then why haven’t they done better trials themselves?”
I often ask patients when I lecture whether they would prefer a benzodiazepine or a neuroleptic next time they become admitted with an acute psychosis. No one so far has preferred a neuroleptic. This gives food for thought.
- Gøtzsche PC. Deadly psychiatry and organised denial. Copenhagen: People’s Press; 2015.
After this, over a year passed, and I still did not hear from the Cochrane Schizophrenia Group. I therefore saw no other option than to send a complaint to Cochrane’s Editor-in-Chief, Karla Soares-Weiser, which I did on 8 December 2022. I explained that my criticism was highly relevant; that the review was misleading; and that the review group had refused to publish my criticism, which it was their obligation to do.
Soares-Weiser replied that she would work with the Cochrane Schizophrenia Group with the aim of getting my comment published in early January 2023 and that she would let me know once this was completed.
Three months later, I checked the review and noted that, although my comment had been published, there was no reply to it in the review, and nothing had been changed in the seriously misleading abstract.
On 6 March, I therefore asked Soares-Weiser for further help and told her that “The whole idea of post-publication criticism of Cochrane reviews has always been that this would help make the reviews better and less biased. ‘Trusted evidence’ is Cochrane’s motto but this abstract, and the fact that the authors do not quote Dold’s review, means that the review is far from being trusted evidence.”
On 25 May, John Hilton, Head of Content Publication and Policies, Cochrane Central Executive, wrote to me that the review has been amended and a response published.
This was four years after I alerted the first author and the Cochrane Schizophrenia Group to the fact that they had published a seriously misleading abstract. The only time I ever heard from the group was when—ironically—they, four years earlier, replied that the group would respond “as soon as possible.”
The response from the “Editorial base Cochrane Schizophrenia” within the review was:
“We would like to thank Prof Peter Gøtzsche for his comments. We agree with him that sometimes the phrase can be misinterpreted as suggested. We have amended the sentence as follows. ‘For the outcome of sedation by 16 hours, there was no difference between haloperidol and benzodiazepine (n = 434, 8 RCTs, RR 1.13, 95% CI 0.83 to 1.54, low quality evidence).’
Whilst the Dold review (Dold M, Li C, Tardy M, et al. Benzodiazepines for schizophrenia. Cochrane Database Syst Rev 2012;11:CD006391) found that there was desired sedation significantly more in the benzodiazepine group compared to antipsychotic monotherapy, this outcome was measured at 20 and 40 min compared to the 16-hour data in this review.
Conducting trials is quite a complex process requiring funding from competitive grants. Whilst it would be good to conduct good quality trials, lack of resources may be a contributory factor as to why more trials in this area have not come forth.”
I have three issues with this response. First, the editors downgrade their mistake by saying that “sometimes the phrase can be misinterpreted.” No, as it was written, it would always be misinterpreted.
Second, they do not find it relevant to comment on the Cochrane review that found faster sedation with benzodiazepines than with neuroleptics arguing that it only assessed the acute effect. This is a nonsense argument. The review I criticised is about treatments for psychosis-induced aggression or agitation, which are acute conditions. It is therefore highly relevant to quote Dold et al.’s Cochrane review.
Third, comparing benzodiazepines with neuroleptics for acute psychosis is not a question of funding. Pragmatic trials can be carried out very cheaply, and blinded drugs are not needed. One can use blinded observers instead and also let the patients evaluate the effect, which is what is most important.
What this deplorable saga illustrates is that Cochrane mental health groups are so keen to protect the many false ideas they share about psychiatric drugs that they are willing to sacrifice scientific honesty and the patients in order to protect the psychiatric guild. I have explained on Mad in America that this also was the case when I tried to publish a Cochrane review of withdrawal methods to help people come off depression drugs.
Unfortunately, the PubMed abstract for the Cochrane review I criticised is still the same, the misleading one from 2017.
Mad in America hosts blogs by a diverse group of writers. These posts are designed to serve as a public forum for a discussion—broadly speaking—of psychiatry and its treatments. The opinions expressed are the writers’ own.
Mad in America has made some changes to the commenting process. You no longer need to login or create an account on our site to comment. The only information needed is your name, email and comment text. Comments made with an account prior to this change will remain visible on the site.