On the Coming Industrialisation of Exploit Generation with LLMs

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution. Exploitation is a particularly challenging task because it requires low-level program reasoning (e.g., about memory layout), runtime adaptation, and sustained progress over long horizons. Meanwhile, it is inherently dual-use, supporting defensive workflows while lowering the barrier for offense. Despite its importance and diagnostic value, exploitation remains under-evaluated. To address this gap, we introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents. Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit. The benchmark comprises 898 instances sourced from real-world vulnerabilities across three domains, including userspace programs, Google's V8 JavaScript engine, and the Linux kernel. We vary the security protections applied to each instance, isolating their impact on agent performance. All configurations are packaged in reproducible containerized environments. Our evaluation shows that while exploitation remains challenging, frontier models can successfully exploit a non-trivial fraction of vulnerabilities. For example, the strongest configurations are Anthropic's latest model Claude Mythos Preview and OpenAI's GPT-5.5, which produce working exploits for 157 and 120 instances, respectively. Notably, even with widely used defenses enabled, models retain non-trivial success rates. These results establish ExploitGym as an effective testbed for exploitation and highlight the growing cybersecurity risks posed by increasingly capable AI agents.

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution. Exploitation is a particularly challenging task because it requires low-level program reasoning (e.g., about memory layout), runtime adaptation, and sustained progress over long horizons. Meanwhile, it is inherently dual-use, supporting defensive workflows while lowering the barrier for offense. Despite its importance and diagnostic value, exploitation remains under-evaluated. To address this gap, we introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents. Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit. The benchmark comprises 898 instances sourced from real-world vulnerabilities across three domains, including userspace programs, Google's V8 JavaScript engine, and the Linux kernel. We vary the security protections applied to each instance, isolating their impact on agent performance. All configurations are packaged in reproducible containerized environments. Our evaluation shows that while exploitation remains challenging, frontier models can successfully exploit a non-trivial fraction of vulnerabilities. For example, the strongest configurations are Anthropic's latest model Claude Mythos Preview and OpenAI's GPT-5.5, which produce working exploits for 157 and 120 instances, respectively. Notably, even with widely used defenses enabled, models retain non-trivial success rates. These results establish ExploitGym as an effective testbed for exploitation and highlight the growing cybersecurity risks posed by increasingly capable AI agents.

Lee Robinson on Twitter / X

Watch a timelapse of GPT-5.2 building a browser!Both websites start out barely working, and then after millions of lines of code, the browser actually works.Pretty cool experiment with long-running agents. https://t.co/KT6HgHEivA pic.twitter.com/8EPbyih8cU— Lee Robinson (@leerob) January 14, 2026

bellard/mquickjs

Public repository of the Micro QuickJS Javascript Engine

The Agent Harness

A specification for agent behavior that LLM frameworks leave undefined: error handling, context management, tool execution, and state transitions.

Georgi Gerganov on Twitter / X

llama.cpp at 100k starsnow that 90% of the code worldwide is being written by AI agents, I predict that within 3-6 months, 90% of all AI agents will be running locally with llama.cpp 😄Jokes aside, I am going to use this small milestone as an opportunity to reflect a bit on… pic.twitter.com/BpN5VL1bYV— Georgi Gerganov (@ggerganov) March 30, 2026

The Bitter Lesson of LLM Extensions

From ChatGPT Plugins to Agent Skills, a look at how we've been trying (and failing) to extend LLMs for the last three years.

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

Large Language Model (LLM) agents have been widely adopted in modern software development workflows. SWE-bench [13] and related works [23, 24, 22, 25, 15] establish the task of issue resolution as a de-facto standard for assessing their capability and usefulness. In this setting, an agent is given an entire codebase, a task description (e.g., a bug report or feature request) in natural language and is instructed to produce a code patch that resolves the issue and passes the repository’s test suite. These benchmarks have been instrumental in demonstrating both the substantial potential and the persistent limitations of current models as SWE agents.

This Web Tool Sabotages AI Chatbots By Making Them Really, Really Slow

Artist Sam Lavigne created ‘Slow LLM’ to make people question their dependence on tools like Claude and ChatGPT. Or at least, make them super annoying to use.

Microsoft Build 2026: Securing code, agents, and models across the development lifecycle | Microsoft Security Blog

Discover how Microsoft enables fast, secure AI development with MDASH and new security capabilities.

Code Mode: the better way to use MCP

It turns out we've all been using MCP wrong. Most agents today use MCP by exposing the "tools" directly to the LLM. We tried something different: Convert the MCP tools into a TypeScript API, and then ask an LLM to write code that calls that API. The results are striking.

GitHub - humanlayer/12-factor-agents at sidebar

What are the principles we can use to build LLM-powered software that is actually good enough to put in the hands of production customers? - GitHub - humanlayer/12-factor-agents at sidebar

Josh Miller on Twitter / X

Here's what @browsercompany's AI eng & ML teams are working on for @diabrowser right now:(This is a pitch to come work for us; info at end)🤖 COMPUTER USE – we've built our own bespoke APIs on top of Chromium to optimize latency, accuracy, and cost of computer-using agents.… pic.twitter.com/c989h1TYnQ— Josh Miller (@joshm) August 19, 2025