No. 019: AlphaFold team disbanded, verification as the bottleneck, Tao's ICM caution - The Hallway Track

The Hallway Track No. 019 - How AI is affecting the practice of science

Verification in the Age of AI Scientists

AI Scientists now operate at a scale that outpaces human capacity for manual review. Systems such as Sakana’s AI Scientist write entire workshop papers end-to-end. Lila Sciences runs autonomous ``AI Science Factories’’ that hypothesize, experiment, and iterate without human guidance. FutureHouse’s Kosmos and Robin generate thousands of candidate hypotheses in a single run, and Google’s Co-Scientist proposes testable experiments at a rate no laboratory can fully evaluate. Each of these systems emphasizes verified results, yet that standard ranges from near-perfect formal proof in mathematics to decade-long clinical trials in medicine, with no shared framework for judging sufficiency across domains. As outputs scale beyond what humans can manually inspect, the question of which results to trust becomes as hard as generating them. The bottleneck for AI for Science is no longer hypothesis generation, it is verification.

No. 022: Palomar proof registry, autonomous protein design, synthetic consensus - The Hallway Track

The Hallway Track No. 022 - How AI is affecting the practice of science

We're not taking the fact-checking powers of AI seriously enough. It's past time to start.

Some notes on the Nobel Prize hallucination that wasn't

How to build an AI scientist: first peer-reviewed paper spills the secrets

AI Scientist, an autonomous research tool, first released in 2024, has now undergone peer review, highlighting its strengths and limitations.

No. 011: Sum-product conjecture disproof, AI-generated authorship fraud, OMB grant-cost rules - The Hallway Track

The Hallway Track No. 011 - How AI is affecting the practice of science

Accelerating Science with Human+AI Review

This issue of NEJM AI features the first two articles published through our accelerated human+AI review process. In this editorial, we describe the invitation-only “Fast Track” process used to revi...

No. 003: Bixonimania hoax, Google's paper-writing agents, AlphaFold expands the frontier - The Hallway Track

The Hallway Track No. 003 - How AI is affecting the practice of science

ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Scientific discovery is defined by the ability to identify the boundaries of existing knowledge and venture into unexplored territory. The ultimate vision for AI in science is problem-driven autonomous research: given a fundamental challenge by a human expert, the AI independently navigates the scientific landscape, uncovers theoretical and empirical bottlenecks, and systematically expands the frontier of knowledge. In this paper, we introduce ScientistTwo, a fully autonomous multi-agent framework designed to realize this vision. Specifically, ScientistTwo takes an initial problem as input, establishes state-of-the-art baselines, formulates novel hypotheses, and coordinates specialized agents to orchestrate an end-to-end discovery cycle without human intervention. Moreover, the framework rigorously conducts experiments using diverse datasets and metrics, refines methodologies through automated ablation studies, and validates research findings via a closed-loop simulated peer-review rebuttal engine. To evaluate ScientistTwo's capabilities against the highest standards of human scientific achievement, we benchmark it across papers accepted at top-tier conferences such as ICLR, ICML, and NeurIPS. As a result, ScientistTwo autonomously generates expert-level, publishable papers and fully verified, executable codebases. Its solutions consistently outperform human state-of-the-art models, and achieve higher average review ratings than human-authored papers under automated AI review agents. These results show that ScientistTwo is not merely an assistive tool but an autonomous scientific pioneer capable of pushing the frontiers of human discovery. Project website: https://scientist-two.github.io/

AI has supercharged scientists—but may have shrunk science

Analysis of 41 million papers finds that although AI expands individual impact, it narrows collective scientific exploration

Mathematician catches AI-generated research error by OpenAI president | Benjamin Golub posted on the topic | LinkedIn

In case you think the problem of science slop is hypothetical, here's something that happened on X yesterday: The president of OpenAI sharing wrong science slop as proud evidence of the power of mathematical AI. Within a day, the mathematician Tony Feng found a gap that undercut the paper's central argument; the author conceded, and (to his credit!) Brockman deleted the tweet as soon as he became aware. Feng was a hero to catch this. The author of the refuted claim is a mathematician but a self-described outsider to this area, with a prior big claim posted and quickly withdrawn. Greg Brockman is not an expert, either. Neither could check what they were amplifying. Heroes catching errors one at a time won't hold against the volume of AI-generated research now coming. We need an automated, independent verification layer for knowledge work, which is what we're building at Refine (refine.ink). Passing such verification should become a necessary step before big claims, especially outside one's own expertise. A link to a bigger-picture post on this is in the comments. | 23 comments on LinkedIn

No. 008: AI co-mathematician, compiler-verified protocols, open infrastructure bets - The Hallway Track

The Hallway Track No. 008 - How AI is affecting the practice of science

No. 014: Science's AI productivity paradox, Sakana clears peer review, AI proof search on open Erdős problems - The Hallway Track

The Hallway Track No. 014 - How AI is affecting the practice of science

The AI Researcher Who Just Quit Anthropic Says It’s ‘Crunch Time for Humanity’

Jacob Coxon talks to WIRED about the “mini Manhattan project” inside Anthropic, the problem with alignment, and why AI labs have just a few years left to make their systems safe.

No. 017: Science EIC contra AI hype, data poisoning of research agents, Vancouver Standard for AI disclosure - The Hallway Track

The Hallway Track No. 017 - How AI is affecting the practice of science

No. 002: Hallucinated citations at scale, ICML catches AI reviewers, scholarly infrastructure gaps - The Hallway Track

The Hallway Track No. 002 - How AI is affecting the practice of science