Doing scientific research is my dream job. Unfortunately, it’s not at all certain that I can keep doing research after getting my PhD degree. Research jobs are scarce and every year the academic job market is flooded with freshly minted PhDs. In practice, this means that only the most prolific PhD students will land a job. In other words, you either ‘publish or perish’. In this blog post I will argue that the culture of ‘publish or perish’, although not a problem in theory, is a problem in practice because of the unfairness of the peer review system. In my view, opening up this system would make it fairer for all researchers, but especially for PhD students.
Based on discussions with colleagues as well as my own experiences I’ve become aware that the peer review system can be random and biased. This intuition is supported by scientific studies of peer review that find that the interrater reliability of reviewers is low, which means that an editor’s (often arbitrary) choice of reviewers plays a big part in whether your manuscript will be accepted (Bornmann, Mutz, & Daniel, 2010; Cicchetti, 1991, Cole, Cole, & Simon, 1981; Jackson, Srinivasan, Rea, Fletcher, & Kravitz, 2011). In addition, studies have found that reviewers are more likely to value manuscripts including positive results (Mahoney, 1977; Emerson et al., 2010) and results consistent with their theoretical viewpoints (Mahoney, 1977). These structural biases as well as the random element make the peer review system unfair as it is unable to consistently distinguish good quality research from bad quality research. This is especially concerning for PhD students who only have a few years to accrue publications to get funding for an academic job. One unfair negative review could nip their career in the bud.
In my view, the solution to the unfairness of the peer review system is straightforward: Switch from a closed peer review system to an open peer review system. Here, I define open peer review as a peer review system in which authors and reviewers are aware of each other’s identity, and review reports are published alongside the relevant article. Ross-Hellauer (2017) found that these two aspects together account for more than 95% of the mentions of ‘open peer review’ in the recent literature. Note that open peer review may also refer to a situation where the wider community can comment on a manuscript, but I do not use that definition here. Below, I list the potential benefits and downsides of switching to an open peer review system.
Potential benefits of open peer review for PhD students
This accountability may cause reviewers to be more conscientious and thorough when reviewing a manuscript. Indeed, a transparent peer review process has been linked to higher-quality reviews in several studies (Kowalczuk et al., 2015; Mehmani, 2016; Walsh, Rooney, Appleby, & Wilkinson, 2000; Wicherts, 2016), although a sequence of studies by Van Rooyen (Van Rooyen, Delamothe, & Evans, 2010; Van Rooyen, Godlee, Evans, Smith, & Black, 1999) failed to find any difference in quality between open and closed reviews. For PhD students higher quality peer reviews are especially important because they are at a stage where feedback on their work is crucially important for their development. Moreover, high quality reviews are fairer for PhD students as such reviews can distinguish more accurately between good and bad research (and thus good and bad PhD students).
McDowell, Knutsen, Graham, Oelker, & Lijkek (2019) found that many PhD students do not find their names on peer review reports submitted to journal editorial staff even though they had co-written the report with a more senior researcher. In such instances of “ghostwriting” the PhD student usually does most of the work while the senior researchers is the only one that profits by gaining appreciation from the editor. An open review system would provide public credit to reviewing PhD students (for example by making reviews citable, Hendricks & Lin, 2017) but would also provide less tangible rewards like senior researchers acknowledging their skills as a high quality scientist (see tweet below).
Of course, this would greatly benefit PhD students because given their status they are likely influenced most severely by scathing or harsh reviews. Indeed, some research shows that reviews are potentially more courteous and constructive when they are open (Bravo, Grimaldo, López-Iñesta, Mehmani, & Squazzoni, 2019; Walsh, Rooney, Appleby, & Wilkinson, 2000).
In a situation with open peer review, journals with no or substandard peer review will be identified quickly and will become known as low-quality journals. Predatory journals can no longer hide behind the closed peer review system and will eventually disappear. This makes life easier for PhD students as it is often difficult to orient the publishing landscape if you are inexperienced with it.
These are often unwarranted citations, but researchers (especially PhD students) are often coerced into adding them because they desperately want to publish their paper. Of course, only researchers who have a reasonable amount of citable papers under their belt would engage in citation manipulation, making it harder for PhD students to compete on the academic job market. Indeed, a prominent case of citation manipulation spurred a group of early career researchers to write an open letter to voice their concern. Open peer review would clearly help here as reviewers thinking of engaging with this unethical practice would think twice if their name and review were public.
For example, it allows PhD students to see how other papers have developed over time or to see that landmark papers have been rejected multiple times before being published. Such insights into the peer review process are very valuable for PhD students as they can get more comfortable with the peer review system and can see that rejections are the norm rather than the exception.
Once a manuscript is rejected it is usually sent out to another journal to undergo a new round of review. It is likely that the arguments used by the first set of reviewers and the second set of reviewers are similar because the first set of reviews was done behind closed doors and authors often change little in between submission. It is estimated that 15 million hours are spent every year by restating arguments while reviewing rejected papers (The AJE Team, 2019). In open peer review, researchers can build on previous reviews, and see the development of the paper, which can free up many hours for valuable research. Of course, not all of the wasted review time is accounted for by PhD students, but because they are likely taking longer than the mean 8.5 hours for a review (Ware, 2008) an open peer review system would be especially time-saving for them.
Potential downsides of open peer review for PhD students
My hope is that I won’t be the one to perish, but the simple fact is that there’s not enough funding available to accommodate every PhD student aspiring a job in academia. That does not need to be a problem as a little academic competition is fine. After all, it only seems fair that the best of the best are tasked with expanding our scientific knowledge. However, the best of the best are only selected as long as the peer review system is fair. Currently, that does not seem to be the case.
In this blog post I have therefore argued for an open peer review system. Implementing this system across the board could increase the quality and tone of peer reviews, could provide PhD students with credit for their reviews, could root out predatory journals, could prevent citation manipulation, could provide PhD students with insight into the mechanics of science, and could lessen the peer review burden for PhD students. Even though the arguments against open peer review should be taken seriously (for example by masking the identities of authors) I am convinced open peer review will create a fairer system. And, as you can see below, the European Journal of Neuroscience, one of the journals that already practices open peer review, wholeheartedly agrees.
Excerpt from the summary report of the European Journal of Neuroscience about their new open peer review system. Retrieved from https://www.wiley.com/network/researchers/being-a-peer-reviewer/transparent-review-at-the-european-journal-of-neuroscience-experiences-one-year-on
Baas, J., & Fennell, C. (2019, May). When peer reviewers go rogue-Estimated prevalence of citation manipulation by reviewers based on the citation patterns of 69,000 reviewers. SSRN Working Paper. Retrieved from https://ssrn.com/abstract=3339568.
Bastian, H. (2018). Signing critical peer reviews & the fear of retaliation: What should we do? https://blogs.plos.org/absolutely-maybe/2018/03/22/signing-critical-peer-reviews-the-fear-of-retaliation-what-should-we-do.
Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PloS ONE, 5(12), e14331.
Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B., & Squazzoni, F. (2019). The effect of publishing peer review reports on referee behavior in five scholarly journals. Nature Communications, 10(1), 322.
Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross- disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119-135.
Cole, S., & Simon, G. A. (1981). Chance and consensus in peer review. Science, 214(4523), 881-886.
Collabra (2019). Editorial Policies. Retrieved from https://www.collabra.org/about/editorialpolicies/#streamlined-review.
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010). Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Archives of Internal Medicine, 170(21), 1934-1939.
Hendricks, G., & Lin, J. (2017). Making peer reviews citable, discoverable, and creditable. Retrieved from https://www.crossref.org/blog/making-peer-reviews-citable-discoverable-and-creditable.
Jackson, J. L., Srinivasan, M., Rea, J., Fletcher, K. E., & Kravitz, R. L. (2011). The validity of peer review in a general medicine journal. PLoS ONE, 6(7), e22475.
Kowalczuk, M. K., Dudbridge, F., Nanda, S., Harriman, S. L., Patel, J., & Moylan, E. C. (2015). Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models. BMJ Open, 5(9), e008707.
Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161-175.
McDowell, G. S., Knutsen, J., Graham, J., Oelker, S. K., & Lijek, R. S. (2019). Co-reviewing and ghostwriting by early career researchers in the peer review of manuscripts. BioRxiv, 617373.
Mehmani, B. (2016). Is open peer review the way forward? Retrieved from https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/is-open-peer-review-the-way-forward.
Ross-Hellauer, T. (2017). What is open peer review? A systematic review. F1000Research, 6. 10.12688/f1000research.11369.2
Ross-Hellauer, T., Deppe, A., & Schmidt, B. (2017). Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLoS ONE, 12(12), e0189311.
Seeber, M., & Bacchelli, A. (2017). Does single blind peer review hinder newcomers? Scientometrics, 113(1), 567-585.
Snodgrass, R. (2006). Single-versus double-blind reviewing: an analysis of the literature. ACM Sigmod Record, 35(3), 8-21.
The AJE Team (2019). Peer Review: How We Found 15 Million Hours of Lost Time. Retrieved from https://www.aje.com/arc/peer-review-process-15-million-hours-lost-time.
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017). Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48), 12708-12713.
Van Rooyen, S., Delamothe, T., & Evans, S. J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: Randomised controlled trial. BMJ, 341, c5729.
Van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1999). Effect of blinding and unmasking on the quality of peer review. Journal of General Internal Medicine, 14(10), 622-624.
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: A randomised controlled trial. The British Journal of Psychiatry, 176(1), 47-51.
Ware, M. (2008). Peer review in scholarly journals: Perspective of the scholarly community–Results from an international study. Information Services & Use, 28(2), 109-112.
Wicherts, J. M. (2016). Peer review quality and transparency of the peer-review process in open access and subscription journals. PLoS ONE, 11(1), e0147913.