A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
OpenAI can’t tell if something was written by AI after all
OpenAI’s tool struggled with accuracy.
OpenCode Zen | A curated set of reliable optimized models for coding agents
OpenCode - The open source coding agent.
Open-world evaluations for measuring frontier AI capabilities
Introducing CRUX, a new project for evaluating AI on long, messy tasks
scaleapi/SWE-bench_Pro-os
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Best LLM for Coding 2026 | AI Coding Model Rankings & Benchmarks
Which AI model writes the best code? We rank every major LLM — open and closed source — across SWE-bench, HumanEval, LiveCodeBench, and Terminal-Bench coding benchmarks. Compare the best LLMs for coding, software engineering, and programming.
Why SWE-bench Verified no longer measures frontier coding capabilities
SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.
AI Coding Agent Benchmarks & Leaderboard | Artificial Analysis
We measure real-world performance of coding agents on software engineering tasks, including cost, token usage, and execution time. We compare how performance changes across agents, models, and execution settings.
AI vs human code gen report: AI code creates 1.7x more issues
We analyzed 470 open-source GitHub pull requests, using CodeRabbit’s structured issue taxonomy and found that AI generated code creates 1.7x more issues.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Large Language Model (LLM) agents have been widely adopted in modern software development workflows. SWE-bench [13] and related works [23, 24, 22, 25, 15] establish the task of issue resolution as a de-facto standard for assessing their capability and usefulness. In this setting, an agent is given an entire codebase, a task description (e.g., a bug report or feature request) in natural language and is instructed to produce a code patch that resolves the issue and passes the repository’s test suite. These benchmarks have been instrumental in demonstrating both the substantial potential and the persistent limitations of current models as SWE agents.
Letting an AI remember tripled its puzzle score - Sensemaker
OpenAI changed two conversation settings, not the model. The result shows why long-running AI tests depend on their memory setup.
Study finds AI tools made open source software developers 19 percent slower
Coders spent more time prompting and reviewing AI generations than they saved on coding.
Can AI Agents Synthesize Scientific Conclusions?
Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions. Yet, their ability to do so in high-stakes domains such as health remains unclear. We introduce SciConBench, a large-scale live benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available. Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.
Agent Skills
AI coding agents take the shortest path to done, which usually means skipping the specs, tests, and reviews that make software reliable at scale. Agent Skill...
What AI coding costs you | Tom Wojcik
What's the effect of the prolonged AI usage among coders and is it tracked correctly, if it all?
