Psychotherapy Studies Frequently Spin Results, Analysis Finds

Randomized trials of psychotherapy often include "spin" to look more positive and use unethical research practices like outcome switching.

By
-
28
3167

Researchers examined 196 trials of psychotherapy for depression. Only a third of the studies found that therapy was helpful. Of those finding therapy unhelpful, 94% reported their results in a misleading manner (known as “spin”) to make the treatment look more helpful than it was.

Additionally, 76% of the studies reported conclusions based on different measures than they had initially planned to use, an unethical practice known as “outcome reporting bias” or “outcome switching.”

“This study shows a high prevalence of protocol discrepancies and spin and low rates of registered trials in psychotherapy research,” the researchers write.

The research was led by Marlene Stoll at University Medical Center Mainz, Germany, and published in the Journal of Clinical Epidemiology. Stoll et al. focused on a single diagnosis, depression, and on trials published between 2015 and 2018, as a way of getting a snapshot of the current state of psychotherapy research for a single, prevalent, well-studied diagnosis.

Of the 196 trials of therapy for depression, a mere 34% found a positive effect of their therapy intervention. 39% found that therapy did not help, and 27% had unclear or mixed results. Worse, 94% of the studies that found therapy unhelpful spun their results to make therapy look better. Only four trials did not spin their results.

What did spin look like?

On average, each paper spun their results about five times (or in five different ways). The most common forms of spin were as follows:

  • The researchers didn’t mention in the abstract that the primary outcome was non-significant (54%)
  • The researchers claimed that the treatment was beneficial despite non-significant findings in the abstract (43%) and the conclusion section (46%)

Another major form of spin that pervaded the papers was distracting the reader by writing about other outcomes before eventually admitting that the primary outcome was non-significant.

Outcome Reporting Bias

Perhaps even more concerningly, 76% of the studies reported outcomes that were inconsistent with their registered trial protocols—which means even the trials that supposedly found a positive effect for therapy may be untrustworthy.

To prevent misleading reporting of results, the World Medical Association’s 2013 Declaration of Helsinki requires that researchers submit a trial protocol before beginning their study. Unfortunately, not every researcher adheres to the Declaration of Helsinki’s guidelines. In this case, 32% of the trials did not have a registered trial protocol.

Registering a trial protocol means that the researcher is telling the reader the specific outcome measures they’re going to use. This protocol remains online (at clinicaltrials.gov, for instance). That way, if researchers report a different set of outcome measures, it can be identified, and the reader knows that the research is untrustworthy.

Why would researchers change their outcome measures? Well, according to Stoll et al., researchers are encouraged to find a positive effect of their treatment. The study sponsor may want them to find a positive effect (this occurs most often in pharmaceutical trials), but researchers are also invested in finding a positive effect themselves. It makes publication a lot more likely, and that’s the biggest metric of a researcher’s career success (and can mean more grant money).

Moreover, researchers are not unbiased. Psychotherapy researchers may be desperate to prove that their mode of therapy is better than the alternative.

So, for instance, researchers who hope to prove that their treatment is effective for depression might use several different depression measures, such as the Hamilton Rating Scale for Depression (HAM-D) or the Beck Depression Inventory (BDI). The more testing you do, the more likely you are to find some difference between groups, just by chance. If the researchers don’t find an effect on the HAM-D, but they do find an effect on the BDI, they may surreptitiously choose only to report the positive finding.

Supposedly, registering trial protocols prevents this practice. But, as Stoll et al. report, 76% of the registered trials still engaged in outcome switching.

What’s the Solution?

Stoll et al. suggest that to prevent these unethical and misleading practices, several steps should be taken.

First, journals must demand that trial protocols be registered, and someone in the editorial process must check for outcome switching.

Second, researcher affiliation must be better understood. That is, we can expect that researchers who support a particular therapy are going to try to make their modality look better.

Third, “registered reports” should be the standard publication strategy moving forward. In this paradigm, researchers submit the methods for their study before conducting the study. The journal then accepts or rejects their paper based on their methods. That way, even if the researchers find a negative result, their study will still be published. This decreases publication bias; currently, only positive results are regularly published.

 

****

Stoll M, Mancini A, Hubenschmid L, Dreimüller N, König J, Cuijpers P, Barth J, & Lieb K. (2020). Discrepancies from registered protocols and spin occurred frequently in randomized psychotherapy trials – a meta-epidemiologic study. Journal of Clinical Epidemiology. DOI: https://doi.org/10.1016/j.jclinepi.2020.08.013. (Link)

RELATED ARTICLESMORE FROM AUTHOR

28 COMMENTS

  1. I would strongly advise people read William Epstein’s latest book Psychotherapy and the Social Clinic in the US soothing fictions and his older book the illusion of psychotherapy – he takes apart the best of the studies and shows them to be riddled with issues showing us the industry is not at all as advertised. Evidenced based my arse.

    Report comment

    • At the same time I’m not even sure being “evidence-based” is something psychology should shoot for…human beings are so complex they may not be amenable to the same kind of epistemological methodologies that are used for thing like chemicals, bacteria, and so on.

      Report comment

      • The other problem with “evidence based” is that the only things that get an “evidence base” are the things that are funded, and we know who pretty much decides what gets funded. Hint: it’s not the recipients of the “evidence based therapies.”

        Report comment

  2. Thanks again Peter.

    The main focus of research and data is really to keep one cohesive system alive and functioning.
    Psychiatry is not interested in the reality of data, they are interested in labels and drugs, save
    for a few who not only reject the data, but reject the foundation and evolution of psychiatry.

    Report comment

  3. I would imagine a big part of the problem with the psychological field is that their “bible” is based upon “bullshit,” and it’s “invalid.” And they are likely feeding most of their “depressed” clients off to the psychiatrists, to be neurotoxic poisoned. Although I will say that the most unethical people I’ve ever met in my life are psychologists.

    They are systemic child abuse cover uppers, thus also pedophile aiders, abetters, and empowerers. They want to get their grubby little hands on the best and brightest American children, “to maintain the status quo.” They have a tendency to try to psy-op innocent others, in order to attempt to steal from us. Plus I’m not certain I’ve ever met a psychologist who was not a pathological liar. Well, except for maybe the psychology professors who could not stop gushing about how wonderful my child was, after giving him a psychology award at his college graduation. But none of this really describes an industry filled with ethical “professionals,” quite the opposite. I’m glad my son got his masters degree in a different field.

    Report comment

    • Actually, I should take that back about the psychology professors, not because my child isn’t an uncommonly intelligent, honest, and well adjusted young man. He is, thankfully. But because my son has friends, who were also psychology majors, and none of them were ever taught in college that the DSM “bible” was debunked as “invalid,” in 2013. Despite the fact my son started university in 2013.

      Report comment

  4. When there is little concordance between clients other than their label, how could it make any sense that a single method of “therapy” would be at all useful? The other problem with studying therapy in this way is that these studies are generally short term. These aren’t real significant relationships between people but a transaction between a professional and their client, and which insurance has put strict limits on. For the cost of therapy with insurance, it would make as much sense to put that money toward things that would actually make life easier. Hire a maid to come in once a week, get a massage once a month, go out to a nice dinner with a friend, go to a movie or the theatre. And if you’re on public insurance, it’s very unlikely you’re getting long-term quality therapy, but instead you’re likely getting drugged to the point you no longer care.

    Report comment

  6. A CALL TO ARMS

    I can’t stress enough how pleased I am with the efforts and effects of Robert Whitaker. This, in combination with the arrival of Open Dialogue, eCPR, and Intentional Peer Support, has thrown a first episode scare into our medical model adversaries, or so it seems. As we do more of what we all do, I foresee that a crack may occur in the smug wall of confidence our opponents always maintain between themselves and us. As the scope of NAMI’s Berlin Wall continues to weaken and erode, the rot of desperation will displace it. DJ Jaffe is already a desperate man.

    Report comment

  7. I wasn’t aware that DJ Jaffe had died. It always makes me feel sad anytime I hear of somebody’s passing, and Jaffe isn’t any exception to that rule. I just read Pete Earley’s announcement of Jaffe’s passing on Earley’s website. I was actually stunned to see two quotes of Jaffe’s therein: Jaffe had actually said that the mental health system, “has nothing whatsoever to do with compassion for patients,” and that patients continue to, “suffer,” accordingly.

    Well, all I can say is that Jaffe must have taken a look at my Facebook page at some point, if he knew his two comments above to be true. What is sad, of course, is the fact that his proposals for a remedy weren’t practical at all, and we all know what those were.

    Anyway, spokesman DJ Jaffe may now be gone, but we’re all still alive and we’re still agitating. We’ve still got our Robert Whitaker.

    Report comment

  8. I’m a member of the National Coalition for Mental Health Recovery’s listserv. Recently, there was discussion about how a documentary film about the mental patients’ movement should be made. I posted the following remark:

    ”I can’t allow this discussion about a documentary film about us to go any further without mentioning Richard Cohen. Richard is a contemporary of such luminary founding activists as Judi Chamberlin, Ted Chabasinski, Leonard Frank, Jay Mahler, Sally Zinman, Paul Dorfner, and Howie the Harp, among others. Though I do not any longer remember the title of this film, Richard made a name for himself when he directed a documentary on the treatment of mental patients in police custody.

    Richard, (who was living in California,) had started work on another documentary, this time of the history of the mental patients’ movement, which he initially titled, No More Sorrow, No More Tears. He later retitled it, On Our Own. He was making this film under the auspices of a production company known as Refuge Films, though later on it was produced by the Film Arts Foundation in San Francisco. In 1991, I spoke to someone at the Film Arts Foundation who told me, at that point, Richard needed an additional $20,000 in order to finish the film. I met Richard Cohen at the Alternatives ‘91 Conference which was held in Berkeley, CA that year. I must have told him I had spoken to the Film Arts Foundation. He asked me if I had the money he needed for the film and he seemed personally offended when I told him I didn’t. A number of still promotional photographs Richard had taken from the film were published in Madness Network News, which I once saw. He’s a superb photographer.

    Ted Chabasinski and Sally Zinman both live in the San Francisco Bay Area, and they might know how to contact Richard Cohen. Someone ought to contact them.”

    Report comment

  9. So, the conclusion is that most researchers in this field are liars? This is what I would have expected, though doesn’t change the fact that it is a sad commentary on the field and on modern life in general.

    This reminds me of what happened to Transactional Analysis in the 1970s. Berne set forth a theory that was uniquely interesting and a method that should have been easy to learn. But no one wanted to do it his way. By the early 1980s there were almost no TA practitioners left, and the one I found wanted me to buy his book on his method.

    If each therapist does something different, then you can never test whether any particular therapy works, because there is no standard that any practitioner is bound to adhere to. So you can never find out what – if anything – works. A perfect plan for complete obfuscation and avoidance of the scientific method. This irresponsibility is part of the legacy of this subject.

    Report comment

    • Most people “shade the truth” I think…does that make them “liars”? Maybe.

      Maybe human beings cannot be studied the way objects can, and all a researcher can do is present his or her theory and let people decide if they want to engage it–no more talk of “evidence-based”.

      Report comment

      • The “evidence-based” approach comes from engineering. There, if a bridge design results in a collapsed bridge, you find a better design. It could work in the humanities, but not if researchers treat people the way they do machines.

        Deception is a very basic problem in this society. You can only overcome it by learning how to spot it and then not tolerating it in your relationships. It’s too widespread to do this 100%, but anything in that direction should help.

        If I were a therapist, I would teach my patient the basic concepts, but try to get him to cooperate even if he were skeptical. But I would work with a supervisor who would grade my sessions for technical precision and correct me if I made any errors. I would pick a therapy that had been tested and proven effective, and would seek to deliver it standardly, with no variations.

        This is not the current practice in the field of mental health, and its current overlords seem so bent on failure, that I doubt it ever wil be. Though the therapies I have studied are said to contribute to “mental health,” they have no real resemblance to anything I’ve ever seen done in the field of psychotherapy.

        Report comment

    • 1wayfarer,
      What a scene that woman went through, although even though she is never going to another shrink
      and did not like her “diagnosis”, she thinks they do a lot of good. I wonder for whom.
      It’s a BS system to HAVE to go see a pretend doctor.

      Report comment

  12. Sam,

    Yes, the requirement she had to HAVE to go see the psychologist is quite shocking. The bogus “diagnosis” given for insurance purposes (well, aren’t they all bogus though), isn’t surprising, since it’s business as usual. The requirement for a DSM label is something that catches people by surprise, who unwittingly walk into a shrink’s office, thinking they are just great kind, caring helpers, who would do no harm. But harm they do. I would say they generally do more harm than good. Somehow there needs to be more awareness put out into the public as to the negative and harmful consequences of engaging in the “mental health” system. Hopefully, if more people write about their experiences, it could lead to changes. But the quest for power and money, not truth, are powerful forces in this world, unfortunately.

    Report comment

  13. I’ve been frustrated for many years by my inability as a writer to publicize the existence and history of the mental patients’ movement. For quite a while, American society and the mental illness Establishment within, (via its allies in the media,) have clobbered me with a one-two punch of slander and subsequent censorship. It’s bad enough that these entities have no respect for the sanctity of my life; when I cry out in pain, my screams are muffled. The control, moderation, and limitation of every facet of every patient’s life is absolutist and monolithic in nature.

    I once sent a very nice piece I’d written to the editors of the newsletter put out by the Church of Scientology. I thought, for sure, they’d be willing to publish it considering their Church’s avowed stance against psychiatry. Well, not only did they not publish it, I couldn’t even get them to come to the telephone when I followed up my submission with a phone call. Over the years I’ve submitted to a myriad of publications including the great, the wonderful Rolling Stone Magazine. All have censored me.

    Alright, so maybe there’s a specific reason why editors/publishers won’t publish patient authors who try to publicize what patients are doing. The editors are aware that there are many opportunists out there who, upon learning of the existence of a movement of mental patients, would try to jump on this bandwagon. But still…

    I’ve submitted a couple of times to the editors of MIA. On both occasions, I was told what I had sent wasn’t long enough. I’m a self-taught author who has never taken any courses in journalism or creative writing. Maybe this is why nothing I write is any longer than a vignette of one or two pages. My stories never contain any documentation.

    Report comment

  14. This along with things like weeding out certain participants preemptively, fudging the statistical calculations and publication bias, all mean that the professional literature should be taken with a giant grain of salt.

    Perhaps human beings need to be studied using different methods (or not “studied” at all), or the entire field of mental health needs to be reconceptualized away from its current rationalist-individualist-biomedical paradigm(s).

    Report comment

LEAVE A REPLY