Former journal editors favor AI reviews in 26 of 30 judgments in Harvard study

Former journal editors, unaware they were comparing AI with human feedback, preferred the AI-written scholarly review in 26 of 30 judgments. That finding anchors an October 22 paper by Harvard researchers Gary King and Steven Worthington, who tested AI against paid experts whose reviews already earned unusually high marks from authors and editors.

Starting with a strong human baseline

The researchers expected AI to win in some areas, some of the time. To test it, they used rapidPeer, a service developed at Harvard’s Institute for Quantitative Social Science that gives faculty feedback on drafts before formal journal submission.

Its reviewers were chosen by field experts from outside the authors’ institutions, paid for their time and given an incentive to earn future work. Instructions emphasized reading the whole manuscript and offering constructive suggestions, rather than deciding whether it met a particular journal’s publication standard.

That approach received strong feedback: 88% of 74 authors surveyed a week after delivery rated their review as high or very high quality. The study included every eligible rapidPeer submission: 124 papers and 134 human reviews collected between 2019 and 2026. Ten papers had two reviews. The manuscripts covered statistical analysis, formal mathematical modeling and qualitative research.

Two ways to generate the competing reviews

For each submission, the researchers used Claude Opus 4.8 at its maximum reasoning setting. Their research design separated two conditions, producing one review per paper in each:

• Matched: Claude received the anonymized manuscript, including figures and tables, and the exact rapidPeer reviewer instructions. It had no external tools.
• Tuned: Claude received a purpose-built prompt and web retrieval. The prompt was written before judging, was not adjusted against judges’ scores and did not contain the judging rubric.

Even the matched condition was not identical to human reviewing. People could consult supplementary files, while the AI’s training data might include work published after a manuscript’s submission. The authors identify both differences in their design.

GPT-5.5, from a different model family, judged the reviews. For overall rankings, it saw the reviews alone; for topic-level comparisons, it also had the manuscript text. Each comparison ran in both presentation orders. A win counted only if the same review won both times.

Winning feedback is not yet better research

The preference extended across all seven review areas assessed by the AI judge, including methods, weaknesses, sources and communication. AI win rates ranged from 92% to 100% among decided comparisons. The authors also report no fabricated sources in the AI reviews and about one-third the incorrect-claim rate found in human reviews.

The researchers acknowledge that AI judges can favor AI-written text or reward fluent but generic feedback. Their human check addressed that concern, but it was smaller: three former editors reviewed 10 pairs, without knowing which review was machine-written. They ranked overall quality, not each review area separately.

The study did not measure whether authors’ subsequent revisions improved their papers, or the collective consequences of switching to AI reviewing. Its task was constructive feedback before journal submission, rather than a journal’s accept-or-reject decision. Those boundaries leave a consequential next step unresolved: whether reviews that judges prefer translate into stronger published research.