AI-assisted Reviewing is Necessary and Should be Open

Peer review is facing a death spiral. AI production tools are speeding it up. AI-assisted reviewing is necessary and should be open.

AI-assisted Reviewing is Necessary and Should be Open

Peer review is facing a death spiral. AI production tools are speeding it up. AI-assisted reviewing is necessary and should be open.

AI peer review goes pro

A what we're reading spotlight

AI’s Growing Role as Scientific Peer Reviewer | Stanford HAI

Stanford computer scientist James Zou is exploring how AI can accelerate scientific research and peer review. His finding: AI excels at spotting gaps, but judgment calls still need humans.

OpenAccess.ai — Rigorous Open Access Publishing

$20 to submit, free to read. AI peer review. Open to human and machine authors. All articles CC-BY 4.0.

How do authors want to use AI for review?

A survey of researchers who compared AI-generated scientific reviews with journal-agnostic human peer review reveals that they overwhelmingly prefer using AI as a self-checking tool before submission rather than as a replacement for human reviewers. It encourages an “author-centric” model in which AI helps researchers improve their manuscripts before they are reviewed by their peers.

Major AI conference flooded with peer reviews written fully by AI

Nature - Controversy has erupted after 21% of manuscript reviews for an international AI conference were found to be generated by artificial intelligence.

ChicagoHAI/OpenAIReview

AI reviewing paper drafts for improvement.

Can AI agents conduct open-ended AI research? Early evidence from two case studies

Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.

Open source was not ready for AI-speed contributions

AI did not create the maintainer burden problem in open source. It accelerated it. Contributors are being amplified, but maintainers are still the verification bottleneck.

AI reviewers are here — we are not ready

Nature - Artificial intelligence promises rapid and polite feedback on papers — but we must first review the reviewer.

Measuring the Impact of Early-2025 AI on Experienced Open-Source...

Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the...

AI-powered Literature Review & Synthesis

Recording from my workshop last week on smarter literature review with AI