A sort of review of the benchmarking industry and what's missing.
When benchmarks go bad - what I learned from measuring performance wrong - Holly Cummins
The world of performance analysis is littered with flawed claims, cognitive biases, dangerous intuitions, and beguiling fallacies. Sadly…
Alex Cheema on Twitter / X
This is why we need open benchmarks for local AI.Otherwise it turns into tribalism and name calling.We will be publishing the largest database of open benchmarks for local AI, tested on 1,000+ real hardware setups. Every device, every interconnect, different… https://t.co/ZsU3PCdSsZ— Alex Cheema (@alexocheema) March 9, 2026
Why it’s getting harder to measure AI performance
The most famous chart in AI might be obsolete soon.
AIベンチマーク「Intelligence Index v5」発表、2つの新しいベンチマークと3つのアップグレード
Artificial AnalysisはAIモデルの性能評価を行っている独立機関で、複数のベンチマーク結果をもとにAIの総合性能を算出する「Artificial Analysis Intelligence Index」は新規モデルの性能を把握するための指標として広く用いられています。そんなArtificial Analysis Intelligence Indexのバージョン5が発表されました。
Announcing Artificial Analysis Intelligence Index v5
Index v5 features 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. 🚀 An ongoing community effort out of Stanford University & Laude… | Steven Dillmann | 13 comments
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. 🚀 An ongoing community effort out of Stanford University & Laude Institute, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. 🧪 ⚛️ 🌍 ➕ ⚙️ Terminal-Bench-Science 0.1 has 70 tasks across the life, physical, earth, mathematical, and engineering sciences. Each one is contributed by a researcher porting their own workflows into challenging agentic tasks, on problems they actually care about. 🧠 Anthropic's Opus 5 dominates most scientific domains with an overall pass rate of ~30%, except the mathematical sciences, where Fable 5 and OpenAI's GPT-5.6 Sol take the lead. SpaceXAI's Grok 4.6 is one of the best models for engineering tasks, Z.ai's GLM 5.3 is the best open model, and Kimi (Moonshot AI)'s Kimi K3 is the only model beyond Opus 5 that lands on both cost and token efficiency Pareto frontiers. 🏆 All leaderboard results are live here: https://lnkd.in/gYh99UfU 💻 You can find all tasks on our Github: https://lnkd.in/gejKmXpj Special thanks to our amazing reviewer team Allen Hart, Ivan Bercovich, Joe Janssen, Jiaming H., Steffen Bollmann, Sergey Aganezov, Ben Blaiszik, Karl Krauth, Aaron Feller, Ben Blaiszik, Yiqi Liu, Han Hou, Robert Joseph G., Robert Zhang; our AI research advisors Ryan Marten, Alex Shaw, Lin Shi, Ben Feuer, Mike Merrill, Gian Segato, Jenia Jitsev, Bodhisattwa Majumder, Peter Clark, Thomas Wolf, Braden Hancock, Andy Konwinski, Alex Dimakis; and especially to our co-leads Sanmi Koyejo and Ludwig Schmidt. | 13 comments on LinkedIn
BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
AI models are increasingly prevalent in high-stakes environments, necessitating thorough assessment of their capabilities and risks. Benchmarks are popular for measuring these attributes and for comparing model performance, tracking progress, and identifying weaknesses in foundation and non-foundation models. They can inform model selection for downstream tasks and influence policy initiatives. However, not all benchmarks are the same: their quality depends on their design and usability. In this paper, we develop an assessment framework considering 46 best practices across an AI benchmark's lifecycle and evaluate 24 AI benchmarks against it. We find that there exist large quality differences and that commonly used benchmarks suffer from significant issues. We further find that most benchmarks do not report statistical significance of their results nor allow for their results to be easily replicated. To support benchmark developers in aligning with best practices, we provide a checklist for minimum quality assurance based on our assessment. We also develop a living repository of benchmark assessments to support benchmark comparability, accessible at betterbench.stanford.edu.
AI Coding Agent Benchmarks & Leaderboard | Artificial Analysis
We measure real-world performance of coding agents on software engineering tasks, including cost, token usage, and execution time. We compare how performance changes across agents, models, and execution settings.
TERMINAL-BENCH-SCIENCE
A benchmark for evaluating AI agents on research workflows across scientific domains
No. 020: Astra solves ten open math problems, AI agents fail at open-ended research, peer review under strain - The Hallway Track
The Hallway Track No. 020 - How AI is affecting the practice of science
What does OSWorld tell us about AI’s ability to use computers?
We review OSWorld, a prominent computer use benchmark. Tasks are relatively simple, many don’t require GUIs, and success often hinges on interpreting ambiguous instructions. The benchmark is also not stable over time.
How Pew Research Center is – and is not – using AI in our work
In this post, we’ll share our current guidelines for the internal use of artificial intelligence and potential areas of experimentation.
Human scientists trounce the best AI agents on complex tasks
State-of-the-industry report finds that, despite the limitations of artificial-intelligence systems, researchers have embraced them.
AI Promised Less Work. I Have Never Worked More. - Orestis Ioannou
Agents made implementation cheap, so I started doing far more of it in parallel. My GitHub history since 2019 shows what that did: as VP of Technology I am back at my full-stack engineer pace, with bursts, burnout and a low season that is no longer low.
