A new meta-scientific review published in a special edition of the Journal of Abnormal Psychology evaluates the evidence behind empirically supported treatments (ESTs) for mental disorders. The authors report finding mixed evidential support, which raises concerns regarding the efficacy of several forms of psychotherapy.
Led by John Kitchener Sakaluk of the University of Victoria and Alexander Williams of the University of Kansas, the researchers scrutinize the statistical evidence base underlying ESTs. They report that most ESTs have low power and replicability, with some of them demonstrating consistently poor performance across various metrics.
“Our analyses indicated that power and replicability estimates were concerningly low across almost all ESTs, and individually, some ESTs scored poorly across multiple metrics, with Strong ESTs failing to continuously outperform their Modest counterparts,” they write.
Recently, researchers have documented the replication crisis in psychology, bringing increased scrutiny to psychological research. Researchers have cast doubts on the reliability of classic experiments, which form the foundation of the discipline. Additionally, other studies have questioned the efficacy of psychiatric medications like antidepressants and neuroleptics. Over the past decade, the mental health field has increasingly faced questions about its conventional treatments and outcomes, as service users and others have cast aspersions on its ethical standing and its scientific status. These concerns are reflected in the new efforts being made to improve the autonomy of patients in mental health care.
Psychotherapy itself lacks internal consensus with clinicians practicing over 500 different forms of therapy. Newer studies have shown that psychotherapy is less effective for those who are poor, have minority status, or are on antidepressants. Researchers have also questioned the ethical foundations and priorities of psychotherapy, with some critics wondering if it can ever escape its neoliberal roots whereby people are treated as entrepreneurial agents.
The authors of this review write that while questions about the efficacy of psychotherapy are of great significance, controlled trials that test this efficacy only started in the 1970s. In 1995, the American Psychological Association’s (APA) Division 12 Taskforce took the responsibility of creating and updating a list of therapies (now known as ESTs). The list was based on repeated controlled trials that were presumed to produce strong evidential support for that treatment. A few examples of some of these empirically supported treatments (ESTs) are Cognitive Adaptation Training for Schizophrenia, Interpersonal Psychotherapy for Bulimia Nervosa, and Psychological Debriefing for Post-Traumatic Stress Disorder, etc. EST approaches are often considered the gold standard for psychotherapy.
Traditional criteria for evaluating the efficacy of psychotherapies depends largely on statistical significance. Therapies that consistently show statistical significance over placebo, no treatment, or another treatment are classified by the APA as having a ‘Strong’ evidence base. Those classified as ‘Moderate’ have shown statistical significance at least once, and the ones considered ‘Controversial’ have demonstrated inconsistent results.
The authors contend that using statistical significance to measure the value of evidence can be problematic. They point to the controversy surrounding psychology’s replication crisis and suggest that the misuse of and misunderstandings surrounding null hypothesis significance testing has led to many of these problems. They state that almost 50% of psychological research contains at least one reporting error, while around 13% have a “gross” reporting error.
Other researchers have raised similar concerns citing that statistical significance does not always translate to clinical significance. This crisis is not restricted to experimental or social psychology but is also influencing clinical literature in medicine and psychiatry.
Additionally, the power of the tests that are used for statistical analysis is also under question. Power, which refers to whether a test can detect an effect when that effect actually exists, has been recommended to be at 80%. Still, reviews report that average power in social science studies is around 44%. As a result, the authors suggest numerous other metrics that cumulatively might create a more rigorous evidential base for ESTs.
This meta-scientific review investigates the evidential value of research that informs 79 different ESTs listed by APA’s Divisions 12. The primary concerns were to examine the evidential value underlying ESTs, mainly focusing on the comparison between the Strong and Moderate ones, and lastly, looking for improvements in the standard for evidential support for ESTs over time.
The authors found that a high number of statistical analyses were not fully reported by researchers making it challenging to re-analyze and verify the data they provided. They also found a lack of clarity in which tests the researchers considered to be fundamental or focal to determine the efficacy of an EST. Most importantly, they found inconsistencies in how well different ESTs performed on various metrics. They write:
“A small number of ESTs (e.g., both Cognitive Processing Therapy and Prolonged Exposure for PTSD) scored consistently well across all or most metrics, whereas a larger number of ESTs—including a number classified as Strong (e.g., Behavioral Activation for Depression, Cognitive Remediation for Schizophrenia, Dialectical Behavior Therapy for Borderline Personality Disorder)—performed relatively poorly across most or all of our metrics of evidential value. Low reporting quality in articles for other ESTs (e.g., Cognitive Behavioral Therapy for Insomnia, Family-Based Treatment for Bulimia Nervosa) made it impossible to calculate many metrics.”
The researchers did find that ESTs classified as Strong or Moderate had fewer reporting errors, and that methodological quality-testing efficacy has shown some improvement over time. ESTs like Exposure Therapy for Specific phobias showed strong efficacy across different metrics, while others like Family Psychoeducation for Schizophrenia consistently fared poorly.
They conclude that numerous problems, like unclear and incomplete reporting of statistical data, primary hypothesis, and focal tests by researchers, was a problem across the board. These studies also did not have 80% power (although improvements in power were observed over time) and “yielded replicability estimates that fell below what is currently normative for clinical research.”
The researchers further note that their analysis allows clinicians to quantify and observe the underlying support for different ESTs, which can have important clinical implications. At the same time, they state that answers about which EST has a stronger evidential base are still not clear or straightforward:
“Based on the available evidence, we don’t know if there are differences in the level of empirical support for ESTs, and we don’t know if ESTs offer benefit beyond that of other bona fide psychotherapies in treating patients with specific diagnoses.”
They suggest that both clinicians and patients should be open to changing the type of therapy in the face of negative or non-existent therapeutic results. This is especially critical now that one cannot merely trust an EST to be founded on substantial evidence.
While the statistical evidence behind ESTs is mixed, and there are many fractures in therapeutic schools of thought, studies have shown the importance of empathy in the success of psychotherapy. Similarly, research shows that common factors like the client’s perception of the therapist’s openness and authenticity are of greater significance than using a particular modality.
Given that psychotherapy is usually underfunded and has limited support from health insurance companies, these findings regarding their efficacy are especially significant for those considering treatment options.
Sakaluk, J.K., Williams, A.J., Kilshaw, R.E., & Rhyner, K.T. (2019). Evaluating the Evidential Value of Empirically Supported Psychological Treatments (ESTs): A Meta-Scientific Review. Journal of Abnormal Psychology, 128 (6), 500-309. (Link)