Drugs like ketamine, psilocybin (mushrooms), LSD, and MDMA are at the forefront of a new wave of overhyped treatments for mental health problems that may fail to deliver on their promises, according to a new article by researchers Michael van Elk and Eiko Fried at Leiden University, the Netherlands. They write that psychedelic research is plagued by methodological problems that make the efficacy and safety of these drugs uncertain.
Despite the minimal research and its limitations, the drugs have been hyped as “miracle” drugs, with some, like esketamine, even receiving FDA approval—despite failing to beat placebo in five of its six initial efficacy trials (the sixth trial reached statistical, but not clinical, significance). In fact, last year, researchers wrote that the promotion of ketamine/esketamine treatments poses “a significant risk to the public.”
In their new article, published before peer review on the preprint server PsyArXiv, van Elk and Fried focus on the top 10 methodological problems rampant in psychedelic research, how these issues undermine the evidence base, and how researchers can avoid them in the future.
“These problems threaten internal validity (treatment effects are due to factors unrelated to the treatment), external validity (lack of generalizability), construct validity (an unclear working mechanism), or statistical conclusion validity (conclusions do not follow from the data and methods),” the researchers write.
Worse, they add, most psychedelic studies feature more than one of these problems, which makes the studies far more unreliable: “These problems tend to co-occur in psychedelic studies, strongly limiting conclusions that can be drawn about the safety and efficacy of psychedelic therapy.”
The Easy Problems
According to van Elk and Fried, some of the rampant problems in this literature are relatively easy to solve. For instance:
Lack of control groups. Many psychedelic research studies are small, unblinded, “open-label” studies with no control group. For instance, the authors discuss a 27-person study of psilocybin for depression, in which 60% of the participants were no longer depressed after a year. That sounds good to the reader—but according to van Elk and Fried, other research has shown that more than half of people with depression would recover without treatment within that time frame. Thus, without a comparison group, this number is meaningless. These participants might have improved anyway.
The solution is simple: design your studies to have a control group, whether to a placebo, an active placebo (something that mimics the side effects of the drug, like dizziness/dissociation), or another treatment (comparing psilocybin to therapy or an antidepressant, for example). Since having a control group is standard practice for drug trials, this should not be difficult for researchers.
Spin, outcome switching, and multiple testing. van Elk and Fried write that the conclusions drawn by psychedelic researchers are often not supported by their studies’ data. Sometimes, the researchers spin poor outcomes to look like positive ones. For instance, one ketamine study found that only 2 of the 14 patients had lower suicidal ideation at three months—but the study’s title claimed that ketamine resulted in “sustained” improvement.
Another prevalent issue in psychedelic research is outcome switching, in which the pre-selected primary outcome measures aren’t reported, and new measures are reported in the final paper. This is often done to hide that the drug wasn’t successful on the primary outcome, so the researchers had to try to find a different measure that showed an effect.
In multiple testing, researchers use a wide range of tests, which increases the risk of finding a positive result just by chance. This can be accounted for by raising the bar for finding a statistically significant effect—if researchers choose to do so. But van Elk and Fried found that psychedelic researchers engaged in multiple testing without statistically accounting for it.
The solution to these problems is that journal editors, reviewers, other researchers, and journalists must hold researchers accountable to ensure they engage in good research practices. One avenue is to hold researchers to the primary outcome they registered before conducting the study. This can be enhanced further if journals accept studies based on methodology—before knowing whether the study found an effect.
Financial conflicts of interest. Most of the research on psychedelics is funded by pharmaceutical companies. van Elk and Fried note that previous research has found that researchers with COI are five times more likely to find a positive effect for a drug than researchers without COI. There are several solutions to this, though: including independent experts as researchers in every part of the study design and conduct, as well as having journal reviewers be independent of industry. van Elk and Fried also call for more independent and government research funding to help with this.
Safety and adverse event reporting. As with all drugs, psychedelics carry risks, including withdrawal. But research has shown that the risks often go unreported and unassessed in clinical trials, including serious harm to body systems, as well as the risk of increased suicidality. One of the most striking harms is the risk of sexual abuse during psychedelic therapy, which can be an especially vulnerable process. van Elk and Fried write that safety and risk assessments must be standardized and reported in all studies on psychedelics.
The Moderate Problems
Small samples. Most psychedelic research studies have very few participants. This leads to several problems. One issue is that it makes researchers more likely to find a result by chance. Another issue is that it is difficult to generalize the results of a tiny study to other people. As van Elk and Fried write,
“We would not finance or conduct a poll about who will win the next US presidency in 20 or 50 participants because such samples are not sufficiently informative regarding the general population that we want to learn about.”
Selection bias. The participants in psychedelic research studies may be of a specific mindset. The people recruited to participate in these studies may be people who already have experience with psychedelics or who are curious and open-minded about them—which may not be true of most people with depression. It could increase the placebo effect. Also, many studies have strict criteria—denying entry to people with multiple diagnoses, suicidality, substance abuse, or unusual presentations, all of which are common in actual practice. Researchers often select the patients most likely to respond; the easiest to treat. Thus, it’s unclear if the results would generalize to most patients.
Lack of long-term outcomes. According to van Elk and Fried, most psychedelic studies assess outcomes for a few hours to a few days. If participants get a short-term “high” from the drug or experience, that is reported as a positive effect. But we don’t know what happens after a few days or after they use the drug repeatedly for months or years.
“As a consequence,” they write, “most studies in the psychedelic literature have at best demonstrated short-lived symptom relief, rather than successful treatment—again, contrary to claims popular in this literature. Especially for people who have suffered from mental disorders for decades, showing that some participants feel somewhat less sad or anxious after receiving psychedelic substances is not equivalent to treatment.”
What is the solution to these issues? Extensive, collaborative, multi-site studies. Including many more participants and following them for a longer time. Carefully reporting all of the inclusion and exclusion criteria—and allowing people into the study who are closer to the most commonly seen real-life patients.
The Hard Problems
Breaking blind. Even if there is a placebo control group, it’s often easy for participants to tell whether they received the drug or not. The effects of psychedelics are quite distinctive and recognizable, especially to those who’ve experienced the drug before. Worse, according to van Elk and Fried, researchers rarely formally assess whether and how badly the blind was broken, and it’s generally poorly reported in the final paper too.
Active placebos (mimicking some of the psychedelic effects) can be one solution. But at the very least, they write, researchers need to assess and report whether their participants could tell if they received the drug or not.
The Placebo Effect. If participants know they received the drug, this often enhances the placebo effect. Likewise, if they’ve enjoyed psychedelics before, they may have an enhanced placebo effect. And finally, if they’ve joined the study because they’re excited about the hype around psychedelics, they may have a high placebo response. All of these aspects mean that the response of people in the drug trial is likely far better than it would be for real-life people who have never had psychedelics before or are a little unsure about them.
According to van Elk and Fried, our uncertainty about the drug’s effects should be emphasized to participants and the general public when the studies are reported. The hype around these drugs, despite our lack of evidence for their safety and efficacy, undermines our data as we try to determine their safety and efficacy.
The Therapeutic Mechanism. Finally, van Elk and Fried mention that there is no consensus around what these drugs are doing neurobiologically that could improve mental health problems. There are theories, of course, but no hard evidence. This is tied to the fact that there is also no consensus around the neurobiology of mental health problems.
But the psychedelic experience comes packaged with powerful psychological effects, including rituals, transcendence, and spirituality. It’s unclear how much these effects may play a role in the positive effects found in psychedelic studies. For instance, one study of ayahuasca concluded that the ritual—not the drug—was responsible for the positive effects.
van Elk and Fried write that researchers need to look into these effects, perhaps by studying rituals, breathwork, meditation, etc., without psychedelic drugs to see if they carry the same benefits. Some research already shows that they might.
Ultimately, according to van Elk and Fried, there are existing solutions to many of these problems. For instance, larger samples and control groups are part of accepted research practice. So why haven’t they been implemented in psychedelic research studies?
There is at least one answer—because it benefits Big Pharma to ignore these issues.
“Given the hundreds of millions that pharmaceutical companies invested into this field, it is hard to imagine that clinical trial experts carrying out and writing up the studies are not aware of these issues. Based on this, we cannot avoid the impression that some of these issues are ignored on purpose, because they benefit the people or companies conducting the studies, but not necessarily the people who new treatments are being developed for,” van Elk and Fried write.
van Elk, M., & Fried, E. (2023). History repeating: A roadmap to address common problems in psychedelic science. Preprint from PsyArXiv, 10 Mar 2023. DOI: 10.31234/osf.io/ak6gx PPR: PPR628919 (Link)