Researchers Warn of Major Threats to the Validity of Psychedelic Research

Warning of “history repeating,” researchers list ten problems with psychedelic research that make conclusions about efficacy and safety uncertain.

4
4136

Drugs like ketamine, psilocybin (mushrooms), LSD, and MDMA are at the forefront of a new wave of overhyped treatments for mental health problems that may fail to deliver on their promises, according to a new article by researchers Michael van Elk and Eiko Fried at Leiden University, the Netherlands. They write that psychedelic research is plagued by methodological problems that make the efficacy and safety of these drugs uncertain.

Despite the minimal research and its limitations, the drugs have been hyped as “miracle” drugs, with some, like esketamine, even receiving FDA approval—despite failing to beat placebo in five of its six initial efficacy trials (the sixth trial reached statistical, but not clinical, significance). In fact, last year, researchers wrote that the promotion of ketamine/esketamine treatments poses “a significant risk to the public.”

In their new article, published before peer review on the preprint server PsyArXiv, van Elk and Fried focus on the top 10 methodological problems rampant in psychedelic research, how these issues undermine the evidence base, and how researchers can avoid them in the future.

“These problems threaten internal validity (treatment effects are due to factors unrelated to the treatment), external validity (lack of generalizability), construct validity (an unclear working mechanism), or statistical conclusion validity (conclusions do not follow from the data and methods),” the researchers write.

Worse, they add, most psychedelic studies feature more than one of these problems, which makes the studies far more unreliable: “These problems tend to co-occur in psychedelic studies, strongly limiting conclusions that can be drawn about the safety and efficacy of psychedelic therapy.”

The Easy Problems

According to van Elk and Fried, some of the rampant problems in this literature are relatively easy to solve. For instance:

Lack of control groups. Many psychedelic research studies are small, unblinded, “open-label” studies with no control group. For instance, the authors discuss a 27-person study of psilocybin for depression, in which 60% of the participants were no longer depressed after a year. That sounds good to the reader—but according to van Elk and Fried, other research has shown that more than half of people with depression would recover without treatment within that time frame. Thus, without a comparison group, this number is meaningless. These participants might have improved anyway.

The solution is simple: design your studies to have a control group, whether to a placebo, an active placebo (something that mimics the side effects of the drug, like dizziness/dissociation), or another treatment (comparing psilocybin to therapy or an antidepressant, for example). Since having a control group is standard practice for drug trials, this should not be difficult for researchers.

Spin, outcome switching, and multiple testing. van Elk and Fried write that the conclusions drawn by psychedelic researchers are often not supported by their studies’ data. Sometimes, the researchers spin poor outcomes to look like positive ones. For instance, one ketamine study found that only 2 of the 14 patients had lower suicidal ideation at three months—but the study’s title claimed that ketamine resulted in “sustained” improvement.

Another prevalent issue in psychedelic research is outcome switching, in which the pre-selected primary outcome measures aren’t reported, and new measures are reported in the final paper. This is often done to hide that the drug wasn’t successful on the primary outcome, so the researchers had to try to find a different measure that showed an effect.

In multiple testing, researchers use a wide range of tests, which increases the risk of finding a positive result just by chance. This can be accounted for by raising the bar for finding a statistically significant effect—if researchers choose to do so. But van Elk and Fried found that psychedelic researchers engaged in multiple testing without statistically accounting for it.

The solution to these problems is that journal editors, reviewers, other researchers, and journalists must hold researchers accountable to ensure they engage in good research practices. One avenue is to hold researchers to the primary outcome they registered before conducting the study. This can be enhanced further if journals accept studies based on methodology—before knowing whether the study found an effect.

Financial conflicts of interest. Most of the research on psychedelics is funded by pharmaceutical companies. van Elk and Fried note that previous research has found that researchers with COI are five times more likely to find a positive effect for a drug than researchers without COI. There are several solutions to this, though: including independent experts as researchers in every part of the study design and conduct, as well as having journal reviewers be independent of industry. van Elk and Fried also call for more independent and government research funding to help with this.

Safety and adverse event reporting. As with all drugs, psychedelics carry risks, including withdrawal. But research has shown that the risks often go unreported and unassessed in clinical trials, including serious harm to body systems, as well as the risk of increased suicidality. One of the most striking harms is the risk of sexual abuse during psychedelic therapy, which can be an especially vulnerable process. van Elk and Fried write that safety and risk assessments must be standardized and reported in all studies on psychedelics.

The Moderate Problems

Small samples. Most psychedelic research studies have very few participants. This leads to several problems. One issue is that it makes researchers more likely to find a result by chance. Another issue is that it is difficult to generalize the results of a tiny study to other people. As van Elk and Fried write,

“We would not finance or conduct a poll about who will win the next US presidency in 20 or 50 participants because such samples are not sufficiently informative regarding the general population that we want to learn about.”

Selection bias. The participants in psychedelic research studies may be of a specific mindset. The people recruited to participate in these studies may be people who already have experience with psychedelics or who are curious and open-minded about them—which may not be true of most people with depression. It could increase the placebo effect. Also, many studies have strict criteria—denying entry to people with multiple diagnoses, suicidality, substance abuse, or unusual presentations, all of which are common in actual practice. Researchers often select the patients most likely to respond; the easiest to treat. Thus, it’s unclear if the results would generalize to most patients.

Lack of long-term outcomes. According to van Elk and Fried, most psychedelic studies assess outcomes for a few hours to a few days. If participants get a short-term “high” from the drug or experience, that is reported as a positive effect. But we don’t know what happens after a few days or after they use the drug repeatedly for months or years.

“As a consequence,” they write, “most studies in the psychedelic literature have at best demonstrated short-lived symptom relief, rather than successful treatment—again, contrary to claims popular in this literature. Especially for people who have suffered from mental disorders for decades, showing that some participants feel somewhat less sad or anxious after receiving psychedelic substances is not equivalent to treatment.”

What is the solution to these issues? Extensive, collaborative, multi-site studies. Including many more participants and following them for a longer time. Carefully reporting all of the inclusion and exclusion criteria—and allowing people into the study who are closer to the most commonly seen real-life patients.

The Hard Problems

Breaking blind. Even if there is a placebo control group, it’s often easy for participants to tell whether they received the drug or not. The effects of psychedelics are quite distinctive and recognizable, especially to those who’ve experienced the drug before. Worse, according to van Elk and Fried, researchers rarely formally assess whether and how badly the blind was broken, and it’s generally poorly reported in the final paper too.

Active placebos (mimicking some of the psychedelic effects) can be one solution. But at the very least, they write, researchers need to assess and report whether their participants could tell if they received the drug or not.

The Placebo Effect. If participants know they received the drug, this often enhances the placebo effect. Likewise, if they’ve enjoyed psychedelics before, they may have an enhanced placebo effect. And finally, if they’ve joined the study because they’re excited about the hype around psychedelics, they may have a high placebo response. All of these aspects mean that the response of people in the drug trial is likely far better than it would be for real-life people who have never had psychedelics before or are a little unsure about them.

According to van Elk and Fried, our uncertainty about the drug’s effects should be emphasized to participants and the general public when the studies are reported. The hype around these drugs, despite our lack of evidence for their safety and efficacy, undermines our data as we try to determine their safety and efficacy.

The Therapeutic Mechanism. Finally, van Elk and Fried mention that there is no consensus around what these drugs are doing neurobiologically that could improve mental health problems. There are theories, of course, but no hard evidence. This is tied to the fact that there is also no consensus around the neurobiology of mental health problems.

But the psychedelic experience comes packaged with powerful psychological effects, including rituals, transcendence, and spirituality. It’s unclear how much these effects may play a role in the positive effects found in psychedelic studies. For instance, one study of ayahuasca concluded that the ritual—not the drug—was responsible for the positive effects.

van Elk and Fried write that researchers need to look into these effects, perhaps by studying rituals, breathwork, meditation, etc., without psychedelic drugs to see if they carry the same benefits. Some research already shows that they might.

Ultimately, according to van Elk and Fried, there are existing solutions to many of these problems. For instance, larger samples and control groups are part of accepted research practice. So why haven’t they been implemented in psychedelic research studies?

There is at least one answer—because it benefits Big Pharma to ignore these issues.

“Given the hundreds of millions that pharmaceutical companies invested into this field, it is hard to imagine that clinical trial experts carrying out and writing up the studies are not aware of these issues. Based on this, we cannot avoid the impression that some of these issues are ignored on purpose, because they benefit the people or companies conducting the studies, but not necessarily the people who new treatments are being developed for,” van Elk and Fried write.

 

****

van Elk, M., & Fried, E. (2023). History repeating: A roadmap to address common problems in psychedelic science. Preprint from PsyArXiv, 10 Mar 2023. DOI: 10.31234/osf.io/ak6gx PPR: PPR628919 (Link)

4 COMMENTS

  1. The most important point in this article is that failing to design adequate studies benefits Big Pharma. So there’s no incentive, and it won’t happen. I suspect researchers know that psychedelic psychotherapy is one big scam and improving their studies will just expose this fact.

    The fact that the FDA, at least to some extent, is going along with this farce is also unsurprising.

    Report comment

  2. A new book on this subject, Strange Drugs Make for Strange Bedfellows, examines the relationship between Ernst Junger and Albert Hoffman.. Reviews follow below.

    Psychedelic drugs and LSD in particular are associated with the Left Wing political radicalism and Hippie culture of the 1960s and as promising to usher in a world of peace love and understanding. However, the discovery of the powerful psychedelic drug LSD emerged in the shadow of the Second World War and has from the outset been a substance of interest to individuals of a radically conservative disposition such as Ernst Jnger, the close friend of Albert Hofmann who first synthesised LSD.

    That interest continues in the shape of elements of the present day Radical Right, who mix an interest in pre-war Volkish ideology and Nordic paganism with psychoactive drugs and contemporary right wing political thought. ‘Strange Drugs make for Strange Bedfellows’ examines the promotion of conservative revolutionary thought within the New Age milieu, which includes contemporary psychedelia, and the interest of individuals from the Radical Right in the role of psychoactive drugs in traditional and contemporary Nordic shamanism.

    “The popular view of psychedelics regards these astonishing drugs as agents of positivepersonal and societal transformation, signposting humanity toward the Age of Aquarius. Yetthere has always been an almost wilfully overlooked and sinister nightside to psychedelia’ssun-kissed Eden.

    Alan Piper’s penetrating study delves deep in the murky historical backwatersof fascist thought, taking us on a long, strange, trip from the trenches to contemporary Nordicneo-paganism, where WWI warrior/philosopher Ernst Jnger’s personal and LSD informedrelationship with Albert Hofmann rubs shoulders with arcane occult and right wing beliefs about psychedelics.

    Piper’s view that the qualities and experiences of LSD and other psychedelics suggests they are,perhaps, neutral tools that can be used to inform any philosophy, liberal or conservative will,rightfully, challenge and provoke many readers. Dense, but highly readable, and satisfyinglyreferenced, Strange Drugs make for strange bedfellows, brings fresh depth and perspective tothe history of psychedelic drugs.”

    – Andy Roberts, author of Albion Dreaming: A popular history of LSD in BritainSince the sixties psychedelic culture has typically been viewed, from both within and without, as a fellow-traveller with progressive, environmental, liberal and communitarian politics. But the psychedelic experience has been understood quite differently in other times and places.

    Alan Piper’s sensitive exploration of its role in the thought of Ernst Jnger, a seminal influence on Albert Hoffman and other psychedelic pioneers of the 1950s, shows how it can offer meaning and validation to quite different political ideologies: individualist, reactionary, even fascist. As well as exposing the appropriation of psychedelic culture by modern far-right interests, his analysis raises profound questions about the meaning of the experience and its political dimensions.”

    -Mike Jay author of High Society: Mind-Altering Drugs in History and Culture.”In Strange Drugs Make for Strange Bedfellows, Alan Piper tells the fascinating and challenging story of the unexplored links between the psychedelic movement which emerged in the mid 20th century, and right wing politics.

    “While to many this connection seems counterintuitive, Piper’s well researched and finely argued work, explains how it is that these movements are not only historically intertwined, but also exist in a complex ideological interplay steeped in ideas of nature and mysticism. Instructive and well worth a read.” – Dr Amy Hale, anthropologist and folklorist, author John Michell, Radical Traditionalism, and the Emerging Politics of the Pagan New Right.

    Report comment

  3. Ever hear the one about the shirt imperiled by the very cloth it was woven from? Talk about your tricky 911 rescue propositions.

    Yet what sounds like a tragedy in the making, proves a perfect set up for the surprise happy ending. To snatch victory from the jaws of defeat in the very last moment delivers the maximum payoff dramatically. Just when it seems ‘all hope is lost.’ So the crowd can go that much wilder.

    If that poor rag could be rescued from itself, surely the same can be done for or with, or about (pick ur preposition) – the cause it’s all for?

    Maybe this Trouble In Paradise can come out smelling like a rose after all – just when the nose was almost fatally assailed by the fumes.

    How interesting to read and be told – and doesn’t it just sound so true and good (and… etc) – that Believe It Or Not (editorially adjusting the grammatical case of a key verb):

    “These problems [which] tend to co-occur in psychedelic studies, strongly limit conclusions that can be drawn about the safety and efficacy of psychedelic therapy.”

    No, wrong. Completely and categorically. Guys, c’mon (you’ve been to college haven’t you?)

    Just like a chain is completely broken even when every single link is perfectly intact –

    Except for one.

    That’s all it takes. Now it’s the whole chain that breaks. Not ‘all over’ just at that single all-critical point.

    No wonder the bottom line of this case in point seems to devolve (shake fist – Chuck Heston, PLANET OF THE APES):

    “Damn you Big Pharma! Damn you to hell”

    All due acknowledgment to ‘team play’ routines – and rapidly developing morale situations for psychedelic hopes and dreams just recent years, after so much honey mooning (to see what it has started to become and is now becoming as seasons change).

    But Earth to Elk & Fried (floating in your tin cans):

    No Virginia. Despite wishful thinking (these guys don’t got ruby slippers do they?)

    There are no “limits” on drawing – fallacious conclusions.

    It’s a distinction from the other kind (which you guys know about?).

    To draw VALID conclusions might be ‘limited’ by these ‘problems.’

    But what stand as limiting factors to those – are another rather more free wheeling kind’s springboard to drawing as many the hell invalid conclusions – as anyone’s pointy little ‘heart’ desires.

    Even at a sterling text dump like this PsyArXiv.

    Despite the superpowers of these ‘problems’ to ‘halt’ so ‘strongly’ conclusions – period (wow ‘wheat and chaff’ same thing after all – who knew?) – in this mosh pit.

    How do these guys do it? Not uniquely oh hell no. Right along with the entire ‘company’ to which they would address this (“hey everybody, we’ve got to…” etc)?

    Have Elk and Fried never heard about a thing called “validity” – with its ‘alter ego’ the invalid?

    They seem expert in the latter, albeit only by show, based on what they tell so cluelessly – all unawares. Right there in plain view.

    If they never heard about most basic things I could understand how they sure screwed up their arithmetic, missing such key variables.

    Funny how little things can mean a lot.

    One little number off, multiply it up a few rounds, massage some data – next thing you know, you got whopping miscalculation.

    So much for the brains that God or Darwin whichever did that gave an elk, cooked but not boiled, just fried.

    In an avg day’s walk on a tightrope (working without a net), how many missteps are needed for disaster to have just struck?

    And so as the sun sets slowly in the west, one silly little link in a chain going poof is once again all it has taken, for another whole chain to be completely broken.

    Such a beautiful one too. Junk narrative. But so sweetly sung.

    Like the clearing up ahead, the light at the end of the tunnel – the salvation of the Timothy Leary dream from itself.

    Rescued from its own Cabinet of Dr ‘psychedelic studies’ Caligari.

    Great to have turned that corner. And isn’t it about time? Now ‘we’ can ‘begin’ to ‘turn’ things around.

    Just when I was starting to think it might never again be safe to go back into the ‘psychedelic studies’ water. Last one in’s a rotten egg?

    Report comment

LEAVE A REPLY