A snapshot of research into answering if frontier AI agents can run R&D into AI (which not surprisingly failed apart from "minor findings" and "engineering steps"). The paper lists its TBDs strengths and limits, worth a read before people who don't know theirs jump in arxiv.org/pdf/2607.27191
Aug 3, 2026 at 6:08 PM
Open-world evaluations for measuring frontier AI capabilities
Introducing CRUX, a new project for evaluating AI on long, messy tasks
Home – Next Frontier AI
Next Frontier AI is run by SPRIND – the German Federal Agency for Breakthrough Innovation. Think of SPRIND as a state-backed deep-tech fund and company builder: we deploy public money with a venture mindset, take real technical risk and are explicitly mandated to try things that can fail.
– Next Frontier AI
Next Frontier AI is run by SPRIND – the German Federal Agency for Breakthrough Innovation. Think of SPRIND as a state-backed deep-tech fund and company builder: we deploy public money with a venture mindset, take real technical risk and are explicitly mandated to try things that can fail.
Pacing the Frontier
A statement from over 1000 employees of frontier AI companies
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.
Introduction to Agents
Discover what actually works in AI. Join millions of builders, researchers, and labs evaluating agents, models, and frontier technology through crowdsourced benchmarks, competitions, and hackathons.
📢 New paper: Forecasts of explosive AI progress hinge on AI agents automating AI research. But most evaluations of agents conducting AI research focus on narrow, verifiable tasks. Can AI agents… | Sayash Kapoor
📢 New paper: Forecasts of explosive AI progress hinge on AI agents automating AI research. But most evaluations of agents conducting AI research focus on narrow, verifiable tasks. Can AI agents conduct open-ended research? https://lnkd.in/gfP-q4CD We gave agents research questions from two unpublished papers, six days, and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers. They unambiguously rejected agents' outputs. Agents were fluent at most *engineering* tasks. They conducted serious literature reviews, debugged GPU environments, ran hundreds of experiments, and turned in camera-ready LaTeX without human help. We also found no evidence of reward hacking. If anything, we found the opposite: the agents started with marketable claims and walked them back to negative results as the evidence came in. But neither agent output was close to the bar of a top conference paper. Both papers suffered from similar failures: poor judgment about the bar for an AI paper submitted to a top conference, the lack of creative problem solving and ineffective backtracking, poor awareness of resources, and instruction drift. This research design has many limitations: the small sample size, non-blind reviews, and the reviewers knowing that the work was AI-generated. We also couldn't test Anthropic's strongest model, because Fable 5 is deliberately limited on frontier AI research tasks, so ended up using OpenClaw with Opus 4.8 (extra-high) for our main experiments and Codex with Sol 5.6 (ultra) for a robustness check. But we think the research design is still helpful in assessing AI agents' ability to conduct research, and it is complementary to evaluations on verifiable tasks, as well as blinded reviews of AI outputs. In follow-up studies, we are expanding the set of non-public papers we evaluate. If you are an AI researcher with unpublished papers, we would love to collaborate with you on our next evaluation. Expression of interest: https://lnkd.in/gpeykJea We also release the agent logs and all the code and data, so that others can conduct their own analyses of our results: https://lnkd.in/gJarPAnb Finally, we plan to conduct such evaluations regularly, and are hiring a senior researcher to help lead these efforts. Apply here: https://lnkd.in/erJZdmve I'm grateful for the core team leading this effort: Peter Kirgis, Andrew Schwartz, Stephan Rabanser, and Arvind Narayanan, and to our collaborators who reviewed AI papers, analyzed agents logs, and gave feedback on the paper: David Demitri Africa, Konstantinos V., Viet Nguyen, Dr Toby D. Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Eric (Yue) Ling, Abhishek Shetty, Helen Toner, Gillian K. Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani
Datacurve | The data engine for frontier AI
Custom data for long-horizon reasoning, software engineering, and data science.
Don't grade an AI agent by its answer - Sensemaker
UK AISI found frontier models taking prohibited shortcuts in cyber evaluations, while self-report and written reasoning failed to reveal them reliably.
A Short Guide to Data Strikes and Conscious Data Contribution in the Context of 2026 Frontier AI
Back to the basics of data leverage.
Cline on Twitter / X
Your daily driver for AI coding shouldn't be a black box -- the stakes are too high. You should have confidence that when you spend $20 on frontier models, you're getting $20 in frontier model intelligence.With subscription tools, you never know what's happening:- Is that… pic.twitter.com/lKUVFLMEip— Cline (@cline) July 8, 2025
Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil
Debates On Frontier Artificial Intelligence Governance: The AI Triad
Analytical Paper Optional: All enrolled students have the option of completing a research paper of at least 20-25 pages, with faculty and peer review of a substantially complete draft. This paper can be used to satisfy the analytical paper requirement for J.D. students. Prerequisite: This course is intended for students intending to work in the […]
1/7 Proud to share our paper, accepted as an oral at ICML '26! We highlight how Big Tech’s influence on AI R&D drives damaging outcomes, and what we as researchers can do about it. We also discuss underlying economic causes. See link for paper, and below for a brief summary arxiv.org/abs/2512.03077
1/7 Proud to share our paper, accepted as an oral at ICML '26! We highlight how Big Tech’s influence on AI R&D drives damaging outcomes, and what we as researchers can do about it. We also discuss underlying economic causes. See link for paper, and below for a brief summary arxiv.org/abs/2512.03077
