ReviewBench: An Extensible Framework for Benchmarking Human and AI Manuscript Review

The volume of scientific manuscripts is rising faster than the available pool of expert reviewers, and AI tools are emerging as a possible response, ranging from frontier large language models applied directly to peer review to purpose-built multi-agent systems. Scalable, standardized benchmarks are needed to regularly evaluate how these tools compare to one another and to human reviewers. We present ReviewBench, an open-source, venue-agnostic framework that compares human and AI reviews across structure, alignment with a paper’s major claims, impact, and critique category. We apply ReviewBench to 145,021 review comments from human reviewers, frontier large language models (GPT-5.2, and Gemini 3 Pro), and Reviewer3.com (R3), a multi-agent peer review system. The dataset spans papers in computer science (ICLR 2025, n = 1,000), social science (Nature Human Behaviour, n = 142), and life science (eLife, n = 1,000). Across disciplines, AI reviews are more structured and engage more directly with a paper’s major claims, with R3 more often surfacing consequential comments, defined as comments capable of undermining those claims. When restricting to critical comments, however, human reviewers rank first on consequential rate on more individual papers than any AI source, despite a lower average. We identify a bimodal reviewer distribution with peaks near 0% and 100%, indicating that many reviewers outperform AI on this metric, but a substantial fraction of reviewers near 0% brings the average down. Critique typing demonstrates systematic differences, where humans emphasize contribution and clarity, while AI emphasizes validity, sufficiency, and transparency. Together, these findings argue against framing AI as a replacement for human review and instead support a complementary model in which AI scales technical verification of major claims while human judgment remains essential for evaluating contribution and shaping editorial decisions.

AI in the workplace vs. AI in education

Quoting Dan Willingham: Students should only use AI for things that they already know how to do well.

Supporting Researchers in a Time of Change - Ithaka S+R

As 2027 approaches, senior leaders are facing another year of uncertainty across the research enterprise. Steep declines in applications and admissions of

PhD students who write dissertation with AI less likely to stay in academia

US PhD graduates who used artificial intelligence were more likely to work in industry.

TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. We operationalize experimental research taste as compute efficiency; a Researcher who reaches the same score as an expert human using half the serial experimental compute has twice the experimental taste. Experimental taste thus acts as a multiplier on experimental compute, making it a key input to forecasts of AI progress. TasteVal consists of 8 novel, challenging, open-ended tasks representative of frontier AI R&D. To isolate taste from coding ability, the model under evaluation acts as a Researcher that iteratively designs experiments while a fixed Coder agent implements them and reports their results. The Researcher executes until either the 40 H100 hour or 120 wall-clock hour budgets are exhausted. We recruit 24 human experts, at least 2 per task, and take the best expert attempt per task as the expert baseline. We evaluate 20 models released between 2023 and 2026. The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost. On TasteVal, the compute multiplier of frontier models has doubled approximately every 3.0 months since December 2025 (95% CI 1.7-5.0), up from every 14 months between 2023 and December 2025. Measured by final normalized performance, frontier models show no trend break, doubling every 14.6 months. To keep TasteVal uncontaminated, we do not release the tasks.

Google opens Deep Review AI tool to help researchers catch manuscript errors

Google Research on Oct. 14 released Deep Review, an AI system designed to flag mathematical errors, logical gaps and unsupported claims in technical documents. The public launch follows pilots that provided feedback on more than 15,000 manuscripts, according to Google. Speaking at Google’s Research@ event in Mountain View, California, Rajesh Jayaram, a research scientist at…

Bending the Curve: How AI is Transforming Scientific Discovery

Discover how researchers use LLMs and specialized AI models like AlphaFold as complementary tools to accelerate innovation and expand the scientific frontier.

AHM Statement on OpenAI’s October 6 Release of Mathematical Documents

[This is a guest post by the Association for Human Mathematics, reposted from their statements page. This blog post was initially written in a different file format and converted using AI. — …

Denario Project: Deep Knowledge Artificial Intelligence Agents for Scientific Discovery

We present , an Artificial Intelligence (AI) multiagent system designed to serve as a scientific research assistant. can perform many different tasks, such as generating ideas, checking the literature, developing research plans, writing and executing code, making plots, and drafting a scientific paper. The system has a modular architecture, allowing it to handle specific tasks, such as generating an idea or carrying out near end-to-end scientific analysis using as a deep-research backend (end-to-end except for the novelty assessment, which is intentionally left for the user to inspect). In this work, we describe in detail and its modules, and illustrate its capabilities by presenting multiple AI-generated papers generated by it in many different scientific disciplines such as astrophysics, biology, biophysics, biomedical informatics, chemistry, material science, mathematical physics, medicine, neuroscience, and planetary science. can also produce work that combines ideas from different disciplines, which we illustrate with a single ( N = 1 ) example: a paper that applies methods from quantum physics and machine learning to astrophysical data. We report the evaluations performed on these papers by domain experts, who provided both numerical scores and reviewlike feedback. These papers were scored by domain experts on our team; we report these scores as an indicative self-assessment rather than an independent evaluation, and we flag the resulting conflict of interest explicitly. Most drafts were rated above the midpoint of a 0–10 scale, with several rated 8–9, indicating that the generated papers reach the level a good undergraduate or early graduate student could achieve after weeks or a few months of work. They should be regarded as first drafts that require expert development, not publication-ready papers. We also document an important failure mode in which, when the input data cannot be read, the system may silently generate synthetic data, which we flag explicitly as a key limitation. We then highlight the strengths, weaknesses, and limitations of the current system. Finally, we discuss the ethical implications of AI-driven research and reflect on how such technology relates to the philosophy of science. We publicly release the code at . A demo can also be run directly on the web at , and the full app will be deployed on the cloud.

Carbon-A: Finding genes in known and unknown genomes

A Blog post by Hugging Face Biology Research on Hugging Face

Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks

Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific Agents, an open-source corpus of 503 profession-specific AGENTS.md profiles, with Gemini 3.8 Flash via OpenRouter in the Pi agent harness. We compare matched profiles with four controls: a minimal baseline ("You are a helpful assistant"), the profile's opening role sentence, a generic scientific rigor guide, and a profile from an unrelated domain. Across nine text-based science benchmarks (4,531 sampled questions, 100 matched profiles), 4,488 items completed all five conditions after API-error retries, scored with automated, rule-based grading. The average profile-baseline accuracy difference is -0.6 percentage points (95% bootstrap interval [-1.5, +0.2] across fixed tasks), and no benchmark shows a statistically clear improvement. Matched profiles produced 1.5-2.3 times as many output tokens and cost 2.2-4.5 times more per successful call. On 60 tool-using BioMysteryBench bioinformatics problems (three runs each for baseline and profile), mean solve rates were 46.7% with the profile and 56.7% at baseline, a difference of -10.0 percentage points (95% interval [-16.7, -3.3]) driven by more frequent token- and time-limit stops under the profile. Longer prompts had one unexpected operational advantage: on SuperGPQA, frequent provider API drops left the short baseline with a correct first-pass answer on only 54.0% of items, against 71.6% with the profile. Generic and mismatched prompts were about as reliable, so this gain comes from prompt length or formatting rather than domain expertise. For the tested model and tasks, loading full profession profiles by default does not improve accuracy and costs considerably more; whether selective retrieval of profile sections or open-ended scientific tasks would change this remains to be tested.

International, cross-sector collaboration commits nearly $2 billion to build foundational data for AI models to predict and treat disease

Biohub, DOE, NIH, and partners will generate open, standardized data to train AI models that predict how cells respond to interventions.

Paperclip

Paperclip — search, read, and analyze 11M+ biomedical papers, regulatory documents, clinical trials, and biological databases (UniProt, PDB, ChEMBL) from the command line.

Claude Frontier Academy: $100M to train 10,000 engineers

Claude Frontier Academy trains Frontier Deployed Engineers to the standard of Anthropic’s own — a $100 million commitment to train 10,000 by the end of 2027.

RETRACTED BOOK: Advanced Nanovaccines for Cancer Immunotherapy

This book discusses how nanomedicine can be used for better drug delivery, to overcome resistance and to boost anti-cancer immunity.

‘Cerebral torment’: IOP retracts nearly 100 conference papers for tortured phrases

IOP Publishing has retracted nearly 100 papers from three of its conference proceedings after the articles were flagged for containing tortured phrases.  Tortured phrases are rephrasings and s…