Peer review is widely regarded as essential for advancing scientific research. However, reviewers may be biased by authors' prestige or other characteristics. Double-blind peer review, in which the authors' identities are masked from the reviewers, has been proposed as a way to reduce reviewer bias. Although intuitive, evidence for the effectiveness of double-blind peer review in reducing bias is limited and mixed. Here, we examine the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Representations (ICLR), a top computer science conference that changed its reviewing policy from single-blind peer review to double-blind peer review in 2018. We find that after switching to double-blind review, the scores given to the most prestigious authors significantly decreased. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance decisions significantly. Nevertheless, we show that double-blind peer review may have improved the quality of the selections by limiting other (non-author-prestige) biases. Specifically, papers rejected in the single-blind format are cited more than those rejected under the double-blind format, suggesting that double-blind review better identifies poorer quality papers. Interestingly, an apparently unrelated change - the change of rating scale from 10 to 4 points - likely reduced prestige bias significantly, to an extent that affected papers' acceptance. These results provide some support for the effectiveness of double-blind review in reducing prestige bias, while opening new research directions on the impact of peer review formats.
翻译:然而,审查者可能受到作者的声望或其他特征的偏差。双盲同侪审查,其中提交人的身份被遮掩于审查者的身份被掩盖于审查者的身份,被建议作为减少审查者偏见的一种方法。虽然直观的证据表明,双盲同侪审查在减少偏见方面的效力有限,而且好坏参半。在这里,我们通过分析提交国际学习代表大会(学习代表大会)的5027份文件的同侪审查档案,来审查双盲同侪审查对威信偏见的影响。这是一个高级计算机科学会议,其审查政策从单盲同侪审查改为2018年双盲同侪同侪审查。我们发现,在转而采用双盲同侪审查之后,给予最有声的作者的分数显著下降。然而,由于许多双盲同侪审查超出了接受的门槛,因此没有严重影响到接受文件的决定。然而,我们发现,双盲同侪审查可能会通过限制其他(非作者)的偏差偏见来提高选择的质量。具体地说,在单盲格式中被否决的论文比那些在接受程度上被否定的更低一些,而在接受程度的同级审查中,而明显地指出,从接受程度为接受程度的更低的平级审查则表明,这种分级审查可能降低了较低的等级标准。