Publication Bias is Corrupting the Scientific Record, New Evidence from PM&R

A new study finds that randomized control trials in Physical Medicine and Rehabilitation (PM&R) show significant publication bias.

6
859

Last week, we reported on a new study that found top psychiatric journals are more likely to publish studies with favorable results—and pharma-funded lead authors—independent of study quality. But it’s not just the psychiatry literature that’s corrupted by publication bias and selective outcome reporting. Another new study has found similar results for the rehabilitation field.

The researchers found that only 13% of pre-registered studies actually ended up published, indicating the potential for publication bias. They add that when studies were published, less than half (48%) reported the pre-specified primary outcomes in the abstract; more than a quarter (27.4%) failed to report the pre-specified outcomes anywhere in their study.

Kanako Komukai and Shuhei Fujimoto conducted the study at Shizuoka Graduate University of Public Health in Shizuoka, Japan, and Sho Sugita at Luxem Co, Ltd in Kawasaki City, Japan. The article was published in the journal Archives of Physical Medicine and Rehabilitation. The researchers identified 5,597 pre-specified protocols for randomized controlled trials in rehabilitation registered between 2013 and 2020; a total of 727 ended up published (13%).

Physical medicine and rehabilitation (PM&R), also known as physiatry or rehabilitation medicine, aims to enhance and restore functional ability and quality of life to those with physical impairments or disabilities affecting the brain, spinal cord, nerves, bones, joints, ligaments, muscles, and tendons.

Why did so few studies end up actually being published? The researchers suggest that three possibilities combined to create this end result:

First, the authors may have decided not to publish because they didn’t find a positive result. This could be because the result was inconvenient for their funding source (pharma, for instance), because it was inconsistent with their own beliefs, or because they doubted that any journal would publish a null result.

Second, the authors may have submitted the paper, but the journals decided not to publish it. Again, this could be because the result was inconvenient or inconsistent with their beliefs; it could also be because journals make money with flashy positive results (since that’s what makes people buy the journal and garners press coverage).

Third, study-related factors may have occurred, such as the study being terminated or still ongoing.

They write that all three of these possibilities likely happened to some degree, but none alone were sufficient to explain the results.

Komukai, Sugita, and Fujimoto write that their findings are similar to those in other fields:

  • Obstetrics/gynecology (50% of articles report results consistent with study protocol)
  • Cystic fibrosis/hereditary diseases (39% of articles report results consistent with study protocol).
  • Cardiovascular/rheumatic/gastrointestinal disease (31% of articles report results consistent with study protocol)
  • Surgical treatments (49% of articles report results consistent with study protocol)
  • Orthopedics (55.8% of articles report results consistent with study protocol)

These results are also similar in psychiatry. A study in 2021 found that 70% of articles on schizophrenia and bipolar disorder failed to report their outcomes accurately, for instance. In terms of publication bias, a study of antidepressant drug trials in 2018 found that 98% of the positive studies were published, but only 48% of the negative findings were published.

Additionally, most of the negative studies ended up appearing positive through outcome switching or spin in the abstracts. Although 52 antidepressant trials found the drug failing to beat the placebo, only four studies were published that accurately reported this result.

Thus, publication bias and selective outcome reporting appear to be corrupting the scientific record across many, if not all, medical specialties.

“These findings provide opportunities for authors and peer reviewers to re-evaluate their attitudes toward reporting negative results and intentionally selected outcomes and for readers to learn about the necessity of understanding and examining the paper, including its research protocol,” write Komukai, Sugita, and Fujimoto.

They argue that we cannot trust the published results, especially in the abstract of studies. The only way to accurately understand the results of a study involves comparing the pre-specified outcomes in the study protocol with the reported results. And only after unpublished negative results are also included in the scientific record will we gain a complete understanding of the true efficacy of medical interventions.

 

****

Komukai, K., Sugita, S., & Fujimoto, S. (2023). Publication bias and selective outcome reporting in randomized controlled trials related to rehabilitation: A literature review. Archives of Physical Medicine and Rehabilitation. Published online June 24, 2023. DOI:https://doi.org/10.1016/j.apmr.2023.06.006 (Link)

6 COMMENTS

  1. “The only way to accurately understand the results of a study involves comparing the pre-specified outcomes in the study protocol with the reported results. And only after unpublished negative results are also included in the scientific record will we gain a complete understanding of the true efficacy of medical interventions.” so basically never…

    Without knowing what is harmfull or does not work there can be no therapeutics. That’s one reason I abandoned medicine: Not even with a Supreme Court order one can access the “truth” in clinical research.

    And this great MIA article tells me why: more than HALF of medical research is never published, and that half is crucial to treatment decisions.

    Report comment

    • Yes, studies of this nature challenge the foundation of existing knowledge within the medical community.

      People often underestimate the profound implications of bias in scientific research. During times of illness or suffering, they place their trust in medical professionals, assuming they possess the necessary expertise.

      However, what if these specialists cannot distinguish between harmful and beneficial substances or therapies? If I were a doctor, I would be deeply troubled by the uncertainty of whether my actions help or harm patients.

      In such a situation, what choices does a doctor have when they realize that some of their prior education may not be reliable? Do they admit their uncertainty, or do they attempt to project unwavering confidence?

      Can you assist us in understanding the various motives that influence this scenario? I tend to believe that the weighty responsibilities in the medical field create immense pressure to avoid admitting any shortcomings, as doing so could lead to legal repercussions. Still, I acknowledge that there may be crucial factors beyond deduction that I’m overlooking.

      Report comment

      • Oh. I wrote a long tirade about what saddles medical practice in general but somehow it got lost. TOS stuff that apparently I can’t even mention or repost…

        Basically that’s why I quit medicine.

        “If I were a doctor, I would be deeply troubled by the uncertainty of whether my actions help or harm patients.”

        Yes, that’s why I quit medicine, and I was just an interconsultant, a physician that saw patients to help the primary provider, sometimes a specialist to reach a diagnosis or rule out one. With very sophisticated technology…

        I was not responsible of more care than confirming or ruling out a suspected diagnosis on more objective grounds than most specialists, GPs and core specialists, let alone practitioners of the psy disciplines.

        I don’t remember being requested by the psy practitioners my more objective services, weird, like they did not need anyone else’s opinion. Particularly with a “no abnormality observed”, and I was very thorough.

        And I was really good at it, I scored third in the entrance exam of my speciality and first in my big universitiy’s yearly exam on peformance of trainees in my speciality, every year. I was above the top 1% (hehe). And I was in an institution that didn’t let trainees to actually read and learn, that was for the folks at really prestigious Hospitals analogous to the NIH, and I beat them all, every year…

        I was chief resident of academic affairs only, and got praised publiclly in front of all trainees in my institution precisely because of my achievements. I have a complimentary cup as chief resident by an international colleagiate body of my training speciality!, unrequested by me, I am a simple folk, a townspeople so to speak. As many people probably do.

        And even after all that, I did not got a good outcome. But the analysis of what saddles medicine and psychiatry practice didn’t pass muster. I am not being critical nor complaining, just explaining why I can not in this reply explain further. I tried.

        There are other limitations and omissions I have, justifiable to my eyes not to some others, but might be off topic, and peculiar…I am an unacknowledged victim of persecution, so I have limitations…

        And I am not peddling abandoning current medical practice, I was trying to explain what in my informal knowledge and experience saddles medicine, particularly the practice of psychiatry.

        Report comment

        • Well, that was an interesting personal story with a wise decision and high academic success compared to others. However, it didn’t really explain the motives of those who do not quit being medical experts in cases where they face uncertainty.

          For someone who has quit, they remain a mystery, as you are not in their shoes. Nonetheless, your guesses should have been much more accurate than mine.

          I tend to believe that earlier actions can tie us down with invisible knots because admitting our misfits would always have adverse consequences. This could easily foster a group-wide bond where everyone reinforces each other’s delusions by repeating a story that denies all mistakes and hurts and portrays them in a positive light.

          That’s what I observe when I watch psychiatrists and families using their services, and it has piqued my interest. I would like to understand it better. It seems to consistently prioritize self-protection over not hurting others and reason, as one’s own safety becomes the top priority.

          Because of this, words born from self-protection seem to distort not only the meaning of sentences but also the meaning of individual words used to express good or bad. This results in a strange world where words cannot be trusted, and good is perceived as bad, and bad as good. This distortion isn’t limited to easily identifiable moral terms but extends to medical language, such as “harm,” “symptom,” and “disorder.”

          It’s not easy to see through this distortion. When we seek knowledge, we often rely on words written by others, and if over 90% accept them, it creates significant pressure and bias. This bias may not solely arise from profit-seeking and the scientific method, as the common explanation goes, but also from self-protection.

          Therefore, just as the medical industry has a profit motive and bias, the motives and biases of those who use their research and products as customers are also important. I feel this is an area that has not been deeply researched.

          There is always a conflict of interests among humans, and currently, it seems that medical language and research goals are defined by the objectives of certain groups, which inevitably takes sides in conflicts between people. In this way, a psychiatrist may be forced to harm others while simultaneously trying to convey that it’s not a conflict between groups and not hurting, but the treatment of an illness. This narrative is then used to label some parties in the conflict as mentally ill.

          I have thought about this extensively because when describing depression, mania, and psychosis using neutral, non-medical terms, focusing solely on natural survival and reproduction goals, nervous system self-regulation, and conflicts between people, it paints a rather grotesque picture of our society and human beings.

          Report comment

  2. I have read drug trial studies. They are not very interesting, not riveting, exciting experiences. They are not best selling spy novels!! It’s very tempting to just skip to the end and read the summary and the written conclusions. Which, I’m sure a lot of doctors do, if they read them at all. BUT THAT IS NOT HELPFUL TO THEIR PATIENTS!
    Because even the published POSITIVE studies are misleading. You’ll look at the numbers and there’s really not much difference between the treatment and the placebo but in the summary the conclusion the author makes usually is not reflective of those numbers. Im usually scratching my head and saying to myself—“huh??? Am I missing something? I wouldn’t say this drug was shown to be effective!”
    But if you’re paid to say a treatment is effective, then to do your job, you say it is, or you don’t get paid. It’s as simple as that for people who need their jobs. And let’s face it, 99% of the world need their jobs or they’re screwed. even 1% do not have the luxury of not making sure people do their jobs. How do you think they maintain their 1% status???

    Report comment

    • Yeah, for some practitioners that seems to be true. But in my relatively brief GP practice it’s not necessary. Way back when althoug briefly, but most of the time with a full schedule and even on every odd day a night visit (terrible for my sleep), most of the time, you get requested even if I did say: There is nothing to worry, this is most likely this, because of this, and most likely it will get better in such period. Just let’s be vigilant about this and this, and if this or anything else happens I’ll be there.

      No need to inflate diagnosis or overprescribe, patients appreciate honesty, integrity, attitude, care and commitment that cannot in my lame unexperienced career be taught in med school. I had to bring it there, they didn’t taught me that. Even if sometimes I missed the target.

      Heck! I even got somewhat attacked for being “normal”, just like most of the community, patients, relatives would have reacted to the “care” I was involved in training. Really I was, being decent in some environments can get you not only harassed but mobbed.

      But I was there, not profit driven but patient driven, always respectfull and willing to learn and help my patients.

      Obviously, I did make mistakes, but that’s on me, not on the community and team I worked with. So, I don’t blame them.

      Report comment

LEAVE A REPLY