| Popis: |
Blinded review is often seen as a solution to inequities that appear to arise in merit-based evaluations where particular individuals or groups are favored. Yet, an equitable merit-based evaluation should also be reliable and valid. We evaluated the equity of blinded and unblinded review in a high-stakes field experiment during peer-review of submissions for an academic conference. In the study, each conference submission ($N = 530$) received both double-blind peer-reviews—both reviewer and author identities were withheld—and single-blind peer-reviews—only reviewer identities were withheld. We found both systems exhibited moderate reliability. Yet, this level of reliability meant that the review systems only agreed on 40\% of the top submissions meaning some of the differences observed between review systems comes from noisy human judgment. The comparison also revealed that single-blind reviews favored senior coauthors, while double-blind reviews slightly favored male authors. Neither author characteristics nor review process consistently predicted talk quality or popularity, but single- and double-blind reviews predicted subsequent publication. These results imply any partiality towards particular authors’ characteristics during single-blind review was not based on calibrated beliefs, and there was not strong evidence suggesting reviewers need to know the authors’ characteristics to judge the merit of the work. The results also suggest that the unfettered use of double-blind review may dismiss a certain perspective or type of work. We discuss ways in which blind review may be adapted to create an equitable merit-based evaluation. |
