Last week, we reported on a new study that found top psychiatric journals are more likely to publish studies with favorable results—and pharma-funded lead authors—independent of study quality. But it’s not just the psychiatry literature that’s corrupted by publication bias and selective outcome reporting. Another new study has found similar results for the rehabilitation field.
The researchers found that only 13% of pre-registered studies actually ended up published, indicating the potential for publication bias. They add that when studies were published, less than half (48%) reported the pre-specified primary outcomes in the abstract; more than a quarter (27.4%) failed to report the pre-specified outcomes anywhere in their study.
Kanako Komukai and Shuhei Fujimoto conducted the study at Shizuoka Graduate University of Public Health in Shizuoka, Japan, and Sho Sugita at Luxem Co, Ltd in Kawasaki City, Japan. The article was published in the journal Archives of Physical Medicine and Rehabilitation. The researchers identified 5,597 pre-specified protocols for randomized controlled trials in rehabilitation registered between 2013 and 2020; a total of 727 ended up published (13%).
Why did so few studies end up actually being published? The researchers suggest that three possibilities combined to create this end result:
First, the authors may have decided not to publish because they didn’t find a positive result. This could be because the result was inconvenient for their funding source (pharma, for instance), because it was inconsistent with their own beliefs, or because they doubted that any journal would publish a null result.
Second, the authors may have submitted the paper, but the journals decided not to publish it. Again, this could be because the result was inconvenient or inconsistent with their beliefs; it could also be because journals make money with flashy positive results (since that’s what makes people buy the journal and garners press coverage).
Third, study-related factors may have occurred, such as the study being terminated or still ongoing.
They write that all three of these possibilities likely happened to some degree, but none alone were sufficient to explain the results.
Komukai, Sugita, and Fujimoto write that their findings are similar to those in other fields:
- Obstetrics/gynecology (50% of articles report results consistent with study protocol)
- Cystic fibrosis/hereditary diseases (39% of articles report results consistent with study protocol).
- Cardiovascular/rheumatic/gastrointestinal disease (31% of articles report results consistent with study protocol)
- Surgical treatments (49% of articles report results consistent with study protocol)
- Orthopedics (55.8% of articles report results consistent with study protocol)
These results are also similar in psychiatry. A study in 2021 found that 70% of articles on schizophrenia and bipolar disorder failed to report their outcomes accurately, for instance. In terms of publication bias, a study of antidepressant drug trials in 2018 found that 98% of the positive studies were published, but only 48% of the negative findings were published.
Additionally, most of the negative studies ended up appearing positive through outcome switching or spin in the abstracts. Although 52 antidepressant trials found the drug failing to beat the placebo, only four studies were published that accurately reported this result.
Thus, publication bias and selective outcome reporting appear to be corrupting the scientific record across many, if not all, medical specialties.
“These findings provide opportunities for authors and peer reviewers to re-evaluate their attitudes toward reporting negative results and intentionally selected outcomes and for readers to learn about the necessity of understanding and examining the paper, including its research protocol,” write Komukai, Sugita, and Fujimoto.
They argue that we cannot trust the published results, especially in the abstract of studies. The only way to accurately understand the results of a study involves comparing the pre-specified outcomes in the study protocol with the reported results. And only after unpublished negative results are also included in the scientific record will we gain a complete understanding of the true efficacy of medical interventions.
Komukai, K., Sugita, S., & Fujimoto, S. (2023). Publication bias and selective outcome reporting in randomized controlled trials related to rehabilitation: A literature review. Archives of Physical Medicine and Rehabilitation. Published online June 24, 2023. DOI:https://doi.org/10.1016/j.apmr.2023.06.006 (Link)