A growing body of research shows that bias can play a deleterious role in the scientific review process and that underrepresented groups in particular receive unfair treatment. Such bias undermines science’s central means of advancing through the identification of high-quality science by peer review. It also threatens a core scientific value: that it is the science that matters, not the traits of the people involved. This project studies the fundamental process of peer review to identify how bias may be diminished and how peer review can be improved. Our findings will potentially inform journals across academic disciplines, as well as conferences, prestigious award-granting institutions, and grant-giving bodies. Bias can enter peer review at every decision juncture of a submitted manuscript: at its initial screening, when reviewers make their recommendations, and even when the final decision is made by the editor. Often, these instances of bias unfairly favor authors from majority groups by virtue of their status or self-similarity with reviewers. We propose to determine whether certain editorial and review practices could diminish such bias by investigating all aspects of peer review at an influential open access scientific journal for the biomedical and life sciences, eLife. There, we have unprecedented access to the content of all texts and discussions surrounding ~35K manuscript submissions. We propose developing a variety of measures reflective of social bias, manuscript quality, and the quality of reviews and consultations, and then using those in predictive modeling to identify qualities of scientific review that diminish bias.
The advance of science in the life sciences rests on a robust review process. The three projects we propose conceptualize and measure the substantive qualities of submitted manuscripts, reviews, and peer consultations using state-of-the-art techniques from natural language processing. A central task is to identify whether the substantive qualities of reviews and consultations — particularly dialogue qualities — can create conditions for a fairer scientific process by reducing the influence that social bias may have on decision outcomes. Our study of desk decisions will help us understand whether and how the first view of manuscripts can be fraught with problems and whether expanded consultation beyond the editor improves or worsens those problems. Our study of reviewer recommendations will help us understand the ways bias might creep into the review process and whether discussion among reviewers diminishes or amplifies such bias. Last, our study of the qualities of reviews and consultations will identify the sorts of reviews and discussions that science needs to prevent the intrusion of bias. By studying these features of scientific review, we will learn how scientific objectivity can be protected from the intrusion of implicit social bias as well as the best practices that may assist it.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
