A recent article published in the journal World Psychiatry examines the frequency of selective data reporting for in research on the use of psychotherapy for depression.
Clara Miguel and her co-authors found that only 40% of studies were pre-registered since July 2005. Of those, 17% were found to involve inflated statistical reporting of effectiveness for these treatments. The authors write:
“Only 40% of trials of psychotherapies for depression published between 2015 and 2018 were prospectively registered, and discrepancies between publications and protocols were noted for 76% of registered trials. It is often assumed that such divergences are the result of intentionally favoring statistically significant findings (“selective reporting”). However, discrepancies could be due to other reasons, such as justified protocol amendments, logistic difficulties or carelessness.”
Bias in the reporting of outcomes for psychiatric medication – as well as psychotherapy – is a significant problem.
Controls exist to try to temper this phenomenon, such as pre-registering research protocols. However, many researchers do not pre-register their protocols, and discrepancies between registered protocols and actual publications are common even when they do.
The current study examines the state of selective outcome reporting for psychotherapies addressing depression. The authors surveyed “all randomized trials comparing psychological interventions to control conditions for adult depression which started enrollment after July 1, 2005, when journal registration mandates became widespread.”
They performed statistical analyses of the studies involved, looking at, for example, differences in effect size for treatment of depression between studies that were prospectively registered and ones that were not.
In analyzing discrepancies between registered protocols and published outcomes, the authors note that potential discrepancies fell into the following categories:
- omission of the registered primary outcome (non-reporting)
- addition of new, not registered, primary outcome
- downgrading of registered primary outcome to secondary
- upgrading of secondary registered outcome to primary
- assessment time point changes
- analysis method changes
Of 185 studies that commenced enrollment after July 2005, 142 (77%) were registered. Only 75 (40%) were pre-registered, however.
51 (68%) of the 75 pre-registered trials were determined to be free of selective reporting.
There were discrepancies between registered and published data for 19 of the 75 studies (25%). 13 (17%) of these involved selective reporting.
The effect size for outcomes of studies with selective reporting was –0.81, while the effect size for studies without selective reporting was –0.54.
In summary, the authors note that their research confirms prior investigations into this phenomenon, although previous studies were smaller and less broad. As stated, they found that prospective registration was found in only 40% of studies, even after many academic journals began to require prior registration in order to receive submissions.
Among prospectively registered studies, 25% exhibited discrepancies between reported protocols and published results. In addition, 17% of these showed discrepancies that favored statistical significance.
The authors conclude:
“Though relatively few, trials with selective reporting were associated with considerably larger effectiveness, when combined in a meta-analysis. Effect sizes diverged by a SMD [standardized mean difference] of 0.27 between trials with and without selective reporting. For reference, selective publication of trials of psychotherapies for depression has been associated with differences in effectiveness of 0.327. Trials with non-reporting of registered outcomes or addition of non-registered ones emerged as the main drivers of effect size inflation.
These data suggest that lack of prior registration and discrepancies between registration and publications remain common in trials of psychotherapies for depression, and are associated with an inflation of effect sizes in those trials, contributing to the current uncertainties in assessing the outcomes of psychological interventions.”
****
Miguel, C., Karyotaki, E., Cuijpers, P., & Cristea, I. A. (2021). Selective outcome reporting and the effectiveness of psychotherapies for depression. World Psychiatry, 20(3), 444-445. (Link)
