AI Automation ·
AI reviews scored higher. How much of peer review should journals delegate?
On November 12, PsyPost covered a working paper comparing AI-written reports with expert feedback from Harvard’s rapidPeer pre-submission service. The researchers used 124 anonymized manuscripts and 134 expert reviews. Three former journal editors each assessed ten randomly sampled review pairs, preferring AI in 26 of 30 judgments. A separate AI judge preferred the AI review in all 132 pairs it decided.[1][2]
Problem definition and research question
An editorial board might want to cut review queues or catch consequential errors that referees miss. Better assignment and follow-up may help the first objective. A second reader with specialized knowledge may help the other. Will a more persuasive AI review improve editorial decisions and the manuscript itself? The answer may change once the board includes the cost of protecting unpublished material.
Evidence and causal analysis
The working paper measured a preference between two pre-submission reviews. Its 30 human judgments came from three people assessing the same ten pairs. The study did not measure the downstream quality of revisions or establish accuracy on accept-or-reject decisions. The capacity to spot a textual inconsistency can also differ from the judgment required to appreciate an unfamiliar claim, especially across fields and languages.[1][2]
| Dimension | International evidence | Korean evidence and application |
|---|---|---|
| Materials | Anonymous rapidPeer manuscripts and compensated expert reports [2] | The Society of Korean Linguistics bars AI input of all or part of manuscripts under review [3] |
| Judgment | Former editors and an AI judge compared pairs of reviews [2] | Referees conduct independent reviews and editors follow journal procedures [3] |
| Validation | The benchmark measured preferences over feedback before submission [2] | A future authorized comparison could begin with open or separately consented manuscripts |
Comparison of alternatives
One option keeps confidential submissions under human review and has the editorial office improve assignment and reminders. Recruitment and compensation still cost money, while access remains governed by the current process. Another option asks a university research office to offer parallel AI and human feedback before submission with the author’s permission. Verification of methods and citations, consent management, and storage add work. Manuscripts under review and covered by the Korean society’s ban would stay out of the comparison.[3]
Implementation and evaluation
A pilot could stratify Korean-language examples from public papers and permitted drafts. Field experts would mark the cited evidence, importance, and revision uptake of each comment. The editorial office would track staff time from preparation to verified feedback. The data steward would suspend the trial and revoke access on any unauthorized upload. The trial would also stop if the miss rate for serious errors exceeded a tolerance set in advance against human review.
Conclusion and open questions
If verified drafts improve and checking consumes fewer hours, an optional service before submission could give authors another path to feedback. Whether the benefit extends to unusual methods and less represented languages remains open, as does who should handle disputes over incorrect feedback.