#AI #CyberSecurity #PromptInjection #AIAgents #LLM #Programming #InfoSec #TechSecurity #ClaudeCode #OpenAIMicrosoft has declared 2026 the "Year of the Agent," but are we walking into a security catastrophe? In this technical deep dive, we explore why AI agents—specifically tools like Claude Code—are inherently vulnerable to a "nightmare" security flaw: Indirect Prompt Injection. We discuss the "Original Sin" of the Von Neumann architecture and how modern LLMs have made this problem even worse by completely erasing the distinction between code and data. From leaking private data to facilitating ransomware, we cover what a malicious prompt can force an agent to do and why standard defenses like "prompt classifiers" or "user confirmation pop-ups" are destined to fail. Less technical version suitable for sending to friends and family: https://www.youtube.com/watch?v=TdHg9ee56Iw on my main @InternetOfBugs channel What you’ll learn: Why LLM architecture knowingly exempts itself from decades of security advances. The difference between direct and indirect prompt injection. Why the "Halting Problem" means AI cannot be used to fix prompt injection. A practical (though "pain in the ass") workflow for running code agents safely using Linux and QEMU virtual machines. Chapter Timings 0:00 - The Security Nightmare of AI Agents 0:36 - The "Original Sin" of Computer Architecture 1:24 - 60 Years of Security Mitigations (Go, Rust, & DEP) 2:29 - How AI Architectures Make Memory Safety Worse 3:24 - Understanding LLM Internals: Code vs. Data 4:16 - What is Indirect Prompt Injection? 4:40 - Malicious Agents: Leaks, Malware, and Ransomware 5:43 - A History Lesson: The 1990s Malvertising Parallel 7:34 - Why Current Defenses (Google & OpenAI) Will Fail 9:09 - The Halting Problem and AI Security 9:40 - The Danger of Malicious Prompts in Open Source Code 10:04 - How to Run Agents Safely: My VM Setup Sources in Pinned comment (too long to fit in this box)
39C3 - Agentic ProbLLMs: Exploiting AI Computer-Use and Coding Agents
Prompt Injection Attacks Are Thwarting AI Hacking Agents
“Context bombing” tricks malicious AI agents into shutting down before they can do harm.
Top AI Security Incidents of 2025 Revealed | Adversa AI
Discover how AI systems are being hacked in the wild — from prompt injection to agent abuse — with real breaches, lessons, and defenses in Adversa AI’s 2025 report.
The lethal trifecta for AI agents: private data, untrusted content, and external communication
If you are a user of LLM systems that use tools (you can call them “AI agents” if you like) it is critically important that you understand the risk of …
Fooling AI Agents: Web-Based Indirect Prompt Injection Observed in the Wild
Uncover real-world indirect prompt injection attacks and learn how adversaries weaponize hidden web content to exploit LLMs for high-impact fraud.
New prompt injection papers: Agents Rule of Two and The Attacker Moves Second
Two interesting new papers regarding LLM security and prompt injection came to my attention this weekend. Agents Rule of Two: A Practical Approach to AI Agent Security The first is …
Vulnerability Research Is Cooked
For the last two years, technologists have ominously predicted that AI coding agents will be responsible for a deluge of security vulnerabilities. They were right! Just, not for the reasons they thought.
AI Agent Traps
As autonomous AI agents increasingly navigate the web, they face a novel challenge: the information environment itself. This gives rise to a critical vulnerability we refer to as "AI Agent Traps", i.e. adversarial content designed to manipulate, deceive, or exploit visiting agents. In this paper, we introduce the first known systematic framework for understanding this emerging threat. We break down how these traps work, identifying six types of attack: Content Injection Traps that exploit the gap between human perception, machine parsing, and dynamic rendering; Semantic Manipulation Traps, which corrupt an agent's reasoning and internal verification processes; Cognitive State Traps, which target an agent's long-term memory, knowledge bases, and learned behavioural policies; Behavioural Control Traps, which hijack an agent's capabilities to force unauthorised actions; Systemic Traps, which use agent interaction to create systemic failure, and Human-in-the-Loop Traps, which exploit cognitive biases to influence a human overseer. This research is not specific to any particular agent or model. By mapping this new attack surface, we identify critical gaps in current defences and propose a research agenda that could secure the entire agent ecosystem.
Securing CI/CD in an agentic world: Claude Code Github action case | Microsoft Security Blog
Microsoft Threat Intelligence identified a prompt injection pathway in Claude Code GitHub Action that allowed access to workflow secrets under specific conditions. This research examines the attack chain, responsible disclosure process, Anthropic's mitigation, and guidance for securing AI-powered CI/CD workflows.
CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
AI agents have significant potential to reshape cybersecurity, making a thorough assessment of their capabilities critical. However, existing evaluations fall short, because they are based on small-scale benchmarks and only measure static outcomes, failing to capture the full, dynamic range of real-world security challenges. To address these limitations, we introduce CyberGym, a large-scale benchmark featuring 1,507 real-world vulnerabilities across 188 software projects. Adjustable to different vulnerability analysis settings, CyberGym primarily tasks agents with generating a proof-of-concept test that reproduces a vulnerability, given only its text description and the corresponding codebase. Our extensive evaluation highlights that CyberGym effectively differentiates agents' and models' cybersecurity capabilities. Even the top-performing combinations only achieve a ~20% success rate, demonstrating the overall difficulty of CyberGym. Beyond static benchmarking, we show that CyberGym leads to the discovery of 34 zero-day vulnerabilities and 18 historically incomplete patches. These results underscore that CyberGym is not only a robust benchmark for measuring AI's progress in cybersecurity but also a platform for creating direct, real-world security impact.
Nicholas Carlini - Black-hat LLMs | [un]prompted 2026
What is AI Agent Security Plan 2026? Threats and Strategies Explained
Learn what AI agent security is, understand key threats like prompt injection and tool abuse, core AI security principles, and best practices to secure AI agents.
System prompt injection — CodeQL query help documentation
If user-controlled data is included in a system prompt or the description of tools for an agentic system, an attacker can manipulate the instructions that govern the AI model’s behavior, bypassing intended restrictions and potentially causing sensitive data leaks or unintended operations.
An AI state of the union: We’ve passed the inflection point & dark factories are coming
OWASP GenAI Security Project Releases Top 10 Risks and Mitigations for Agentic AI Security
Culmination of over 100 industry leaders’ input and extensive published resources to deliver critical guidance to address Agentic AI Security risks WILMINGTON, Del. — Dec. 10, 2025 — The OWASP GenAI Security Project (genai.owasp.org), a leading global open-source and expert community dedicated to delivering practical guidance and tools for securing generative and agentic AI, […]
Microsoft Build 2026: Securing code, agents, and models across the development lifecycle | Microsoft Security Blog
Discover how Microsoft enables fast, secure AI development with MDASH and new security capabilities.
