Peer review
2020-06-07 → 2026-11-09
Rebuttal#
Doing a review#
Studies on peer review#
Does peer review work?#
- Big names or big ideas: Do peer-review panels select the best science proposals?
- Zhang et al. (2026) use deidentified submission data from Science and Science Advances to separate editorial-stage selectivity from peer-review dynamics at elite general science journals.
Reviewer matching and algorithms#
- Charlin & Zemel, The Toronto Paper Matching System
- Shah, An Overview of Challenges, Experiments, and Computational Solutions in Peer Review
- Saveski et al., Counterfactual Evaluation of Peer-Review Assignment Policies
AI in peer review#
- Peer review
Incentives and credit#
- Moles et al. (2026) — proposes a transparent universal credit ledger across journals so reviewing is rewarded; relevant to the worsening reviewer shortage and the verification debt argument.
- Gorelick & Clark (2026) — Biology Open’s “Fast & Fair” scheme paid pre-contracted reviewers £220 per report delivered within four working days, cutting time to first decision from ~38 to 5.5 working days without lowering editor-rated quality. Q&A with editor-in-chief Daniel Gorelick, reported by Miryam Naddaf in Nature (2026).
- Pottarath et al. (2026) — MetaArXiv preprint documenting freelance paid reviewing through third-party vendors; the scale and journal clients remain unclear. Nature news coverage.
Peer review protocols#
- Gross & Bergstrom, Why ex post peer review encourages high-risk research while ex ante review discourages it
Peer review biases#
Demonstrations of Bias in peer review:
- Simcoe & Waguespack, Status, Quality, and Attention: What’s in a (Missing) Name?
- Tomkins et al., Reviewer bias in single- versus double-blind peer review
Zhang et al. (2026) add a multistage-journal view: associations with institutional prestige, team size, topic, country, and author gender are generally stronger in editorial review than later peer review.
“Reviewer #2 is not the problem. Reviewer #3 is.”
Experiments#
- 2014 NeurIPS experiment: https://inverseprobability.com/2014/12/16/the-nips-experiment
- http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
- Cortes & Lawrence, Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment
- http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
- Tomkins et al., Reviewer bias in single- versus double-blind peer review — WSDM 2017 experiment
- The review process for NIPS 2016
- The NeurIPS 2021 Consistency Experiment
Replacing journals#
- Brembs et al. (2023) — frames decades of inaction as a social dilemma and proposes a decentralized network of substitutable service providers under scholarly governance in place of journals. See Open access.
History#
- History
Articles#
- The end of Supporting Material? by Sune Lehmann — How can we solve “supplemental material arms race”?
- Peer review and the meat grinder by Aaron Clauset
- http://www.genomesunzipped.org/2011/07/why-publish-science-in-peer-reviewed-journals.php
- Publish-or-perish: Peer review and the corruption of science by David Colquhoun
- Peers, review your actions — don’t review for free, for closed journals.
- The Nastiness Problem in Computer Science by Bertrand Meyer
- Yes, Computer Scientists Are Hypercritical by Jeannette M. Wing
- A world without referees by Larry Wasserman
- Who’s Afraid of Peer Review?
- Reviewing the reviewers
- A little bias in peer review scores can translate into big money, simulation finds
- Let’s make peer review scientific
- It’s time to get rid of the peer-reviewed paper by Juan Pablo Pardo-Guerra — argues that the journal article has calcified into a productivity metric, and that AI-generated papers will worsen the flood unless evaluation practices change.
- Paper gestalt by Carven von Bearnensquash