Peer Review Unfairly Favors Famous Names, Study Finds

Researchers in Austria found that when a scientific paper submitted for peer review had the name of a Nobel Prize laureate attached, reviewers were at least 20% more likely to recommend acceptance compared to when the name was not attached.

The authors conducted an experiment asking reviewers to assess a paper written by laureate V. L. Smith—who won the Nobel Memorial Prize in Economic Sciences in 2002—and a relatively unknown colleague. In total, they received 534 reviews.

The study’s authors varied whether Smith’s name was mentioned, whether the relatively unknown colleague was mentioned, or whether the paper was given anonymously. The results suggest significant publication bias toward prestigious researchers such as Smith.

“Peer review has been the key method for research validation since the first scientific journals appeared some 300 years ago. For researchers—and especially for young scientists, who must excel for scientific advancement—it is crucial that this process be fair and impartial. Merton, however, argued that ‘eminent scientists get disproportionately great credit for their contribution to science while relatively unknown scientists tend to get disproportionately little credit for comparable contributions.’

Alluding to the Gospel according to Matthew 25:29, Merton termed this pattern of the misallocation of credit for scientific contributions ‘the Matthew effect in science,’ while others called it ‘status bias,’” write a team of authors led by Jürgen Huber, Head of the Department of Banking and Finance at Innsbruck University in Austria.

In order to test the possible effect of “status bias” on acceptance of research publications, authors of the current study invited a number of professionals in the field to review an unpublished paper written by the previously mentioned Nobel Prize laureate V.L. Smith (who has over 50,000 Google Scholar citations) along with a relatively unknown colleague (with 42 Google Scholar citations). This paper was first submitted to the Journal of Behavioral and Experimental Finance, after which more than 3,300 potential reviewers were invited to provide peer review.

Acceptance rates varied between the three conditions: low-prominence author mentioned, no author mentioned, and the high-performance author mentioned.

Respectively, 28.52% of potential reviewers accepted the invitation for the low-prominence author, 30.74% for the anonymous author, and 38.54% for the high-performance author. This already shows a bias toward being willing to peer review work from those of higher social status.

Furthermore, the authors hypothesized that when the high-prominence name was attached, there would be a significant increase in acceptance of the article compared to when it was anonymized. Likewise, they hypothesized that the inclusion of the low-prominence author’s name would also decrease acceptance compared to the anonymized version.

As the authors of the current study noted, the most important findings were related to the choices given to those invited for peer review: reject, major revision, minor revision, or accept. These options are standard across multiple scientific disciplines and journals requiring peer review.

Reporting their statistically significant findings, they stated:

Or, framed in the positive:

The authors also included six standardized questions which are common to the peer review process for journals published by Elsevier:

• Was the subject worthy of investigation
• Was the information new
• Were the conclusions supported by the data
• Was the manuscript suitable for the particular journal
• Was the organization of the manuscript appropriate
• Were the tables and figures appropriate

A Likert scale from 1-5 was employed here, one being “strongly disagree” and five being “strongly agree.”

As with previous findings, the authors found an “upward shift” in the assessment of the paper’s quality with these criteria in mind when the high-prominence author’s name was attached, compared to both the anonymized version and the version with the low-prominence author.

The authors noted some limitations to the study. One, when consent for peer review was attained, the authors made it clear as part of their ethical standards that this was not just an ordinary peer review but was also part of a study on the peer review process itself. This may have had unintended effects.

A second limitation was reported in the context of potential racism. The low-prominence author’s last name was Inoua, and the authors state that he has dark skin, while Smith (a more conventionally Caucasian name) has white skin. The authors clearly state that:

Thirdly, the authors are transparent that this paper, and the journal it was submitted to, may not represent the entirety of all scientific disciplines. For example, it is possible that the findings would have been different in a discipline other than business/economics.

While stating that they are “confident that similar effects likely are also present in other fields,” the authors recommend performing replication studies in other scientific disciplines to test these hypotheses further.

Previous research into this area has found similar findings: for example, that the reputation of scholars has a biasing effect on the peer review process even when the reviewers themselves do not have access to the names of authors (as in double-anonymous peer review) because these decisions ultimately rest with journal editors, who can also be prone to bias. Likewise, journal editors who have worked with authors previously are more likely to favorably assess and accept their work in the future.

If these issues are not resolved through some alteration of the peer review process, then it is likely, as the authors suggest, that academics from marginalized groups, as well as “lower-prominence academics” in general, will remain at a significant disadvantage in terms of publication acceptance.

Concluding, the authors do make some suggestions for alternative peer review mechanisms:

****

Huber J., Inoua S., Kerschbamer R., König-Kersting C., Palan S., & Smith V. L. (2022). Nobel and novice: author prominence affects peer review. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2205779119 (Link)