Prominent Academic Journals Score Disturbingly Low on Measures of Transparency and Openness

A new study published in the Journal of the Royal Society of Medicine finds that prominent academic journals score very low on measures of transparency and openness.

Antoni Gardener and colleagues report that although Transparency and Openness Promotion (TOP) scores improved slightly during the COVID-19 pandemic (from an average of 5/24 to 7/24), overall scores are disturbingly low for many established academic journals.

The journals they examined scored lowest in terms of preregistration of study protocols/analysis plans and submission of replication studies. However, they scored highest in adherence to data transparency. The authors write:

The authors set out to assess the TOP scores for prominent academic journals and to examine the impact of the COVID-19 pandemic on transparency and openness practices. To accomplish this goal, the researchers selected 19 journals from google scholar’s top health and medical sciences publications. Then, they examined their transparency and openness policies in February 2020 (pre-pandemic) and May 2021 (post-pandemic).

Two independent study authors measured TOP scores, with discrepancies decided by a third independent researcher. TOP guidelines address eight areas of journal policy: data citation, data transparency, code transparency, materials transparency, design and analysis transparency, study preregistration, analysis preregistration, and replication.

Scores were determined using a scale from 0 to 3 in which 0 means the policy of the journal made no mention of the corresponding category, 1 “discloses” how the study dealt with the category, 2 “requires” the study to address the category, and 3 “verifies” that the study addressed the category. Additionally, the authors measured whether each journal was a “signatory” of TOP (expressing their willingness to adopt TOP standards, no = 0 points, yes = 1), whether they permitted registered reports (no = 0, yes = 2) and whether they used open science badges (no = 0, yes = 2).

The current work also examined conflicts of interest in these journals using the four standards of the International Committee of Medical Journal Editors (ICMJE). These standards are: receiving payments from a 3rd party, having a financial relationship with any entity that could influence research, any patents related to the research, and any other relationships that could impact the research. The current work scored conflict of interest disclosures on a scale of 0 to 4, with 0 disclosing none of these standards and 4 disclosing all.

Regarding the 8 TOP standards, the 19 journals examined in the current work had a median score of 5/24 before the COVID-19 pandemic, improving slightly to 7/24 during the pandemic. 58% of the journals examined showed no change during the current research. 26% improved TOP policies, with about 10% seeing a reduction in their scores during the same period. The highest-scoring Journals had TOP scores of 14/24, with the lowest receiving a score of 0/24.

One additional journal became a signatory to TOP during the course of the current research (9 were signatories before the pandemic, ten after). Only one journal accepted registered reports and did so both before and during the pandemic. 84% of the examined journals disclosed all 4 ICMJE conflict of interest standards before the pandemic, increasing to 95% after.

The authors acknowledge several limitations to the current work. First, TOP standards include some ambiguous language (for example, “should,” “strongly recommend,” and “expect”). Second, the authors examined journal policies and not whether or not the articles published actually upheld these policies. Third, TOP guidelines may be less applicable to highly specialized journals, which the authors chose to exclude from the current study.

The authors conclude:

Many academic journals have demonstrated publication bias in clinical studies, with research that shows statistically significant results most likely to be published. This inflates the perceived efficacy of drugs (such as antipsychotics and antidepressants) and therapies, as studies that find them ineffective often go unpublished. In addition, journals routinely publish their own editors, calling their objectivity into question.

Research has shown extensive undisclosed conflicts of interest in medical research. These extensive conflicts of interest are often linked to “unduly favorable” editorials written by authors with some stake in the outcome of the study they are “evaluating.” The ethical issues surrounding academic journals are so pronounced that one author commented recently: “journals have devolved into information laundering operations for the pharmaceutical industry.”

Academic publishing is a massive industry on par with music and film. Research suggests that, similar to the music and film industry, getting your work published in many academic journals is more about having a big name or deep pockets than producing a sound study.

 

****

Gardener, A. D., Hick, E. J., Jacklin, C., Tan, G., Cashin, A. G., Lee, H., Nunan, D., Toomey, E. C., & Richards, G. C. (2022). Open science and conflict of interest policies of medical and health sciences journals before and during the COVID-19 pandemic: A repeat cross-sectional study. JRSM Open, 13(11), 205427042211321. https://doi.org/10.1177/20542704221132139 (Link)