How Cloudflare uses generative AI to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.
Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf, and allowing targeted blocking of AI crawling on sites that serve ads.
Getting Bots to Respect Boundaries
Getting Bots to Respect Boundaries How AI Crawlers Are Straining Web Infrastructure Audrey Hingle January 2026 Image by Janet Turra & Cambridge Diversity Fund betterimagesofai.org creativecommons.org/licenses/by/4.0 Contents Introduction: Setting up the Problem 3 Understandin...
AI Agent Traps
As autonomous AI agents increasingly navigate the web, they face a novel challenge: the information environment itself. This gives rise to a critical vulnerabil
Content Independence Day: no AI crawl without compensation!
It’s Content Independence Day: Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content.
Ultimate Block List to Stop AI Bots | Perishable Press
More than you might think, AI (Artificial Intelligence) and ML (Machine Learning) bots are crawling your site and scraping your content. They are...
Project Think: building the next generation of AI agents on Cloudflare
Announcing a preview of the next edition of the Agents SDK — from lightweight primitives to a batteries-included platform for AI agents that think, act, and persist.
Track, control, and optimize for AI agents and bots | Known Agents (formerly Dark Visitors)
Use Known Agents to turn the rising wave of AI agents, LLM assistants, and other bots crawling your website into a new growth channel for your business
Cloudflare Workers AI | Open-source AI inference
Workers AI facilitates the scalable development & deployment of AI applications at the edge.
Meet Gatebot - a bot that allows us to sleep
In the past, we’ve spoken about how Cloudflare is architected to sustain the largest DDoS attacks. During traffic surges we spread the traffic across a very large number of edge servers.
The age of agents: cryptographically recognizing agent traffic
Cloudflare now lets websites and bot creators use Web Bot Auth to segment agents from verified bots, making it easier for customers to allow or disallow the many types of user and partner directed bots the explosion of AI agents has created.
Introducing Kitesurf: The agent-first browser that runs in V8 isolates on Cloudflare Workers
We should be giving all agents tools that excel at what’s important for an AI model. Kitesurf is Cloudflare’s new stateless, highly scalable, and cost-effective web browser that runs entirely on top of Workers and was designed specifically for the Agentic Cloud.
Swamp: The right way to build with AI
Swamp is the right way to build with AI: compose repeatable behaviors, not token-hungry agents. Any harness, any API, secrets in vaults. Free for solo use.
Heaps do lie: debugging a memory leak in vLLM. | Mistral AI
The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.
Have it both ways: stay discoverable in search while disallowing AI training
Cloudflare is giving site owners a way to stay discoverable while disallowing AI training. New controls and an Accountable designation establish a shared model with Apple, Google, and Microsoft.
AI Agent Traps
As autonomous AI agents increasingly navigate the web, they face a novel challenge: the information environment itself. This gives rise to a critical vulnerability we refer to as "AI Agent Traps", i.e. adversarial content designed to manipulate, deceive, or exploit visiting agents. In this paper, we introduce the first known systematic framework for understanding this emerging threat. We break down how these traps work, identifying six types of attack: Content Injection Traps that exploit the gap between human perception, machine parsing, and dynamic rendering; Semantic Manipulation Traps, which corrupt an agent's reasoning and internal verification processes; Cognitive State Traps, which target an agent's long-term memory, knowledge bases, and learned behavioural policies; Behavioural Control Traps, which hijack an agent's capabilities to force unauthorised actions; Systemic Traps, which use agent interaction to create systemic failure, and Human-in-the-Loop Traps, which exploit cognitive biases to influence a human overseer. This research is not specific to any particular agent or model. By mapping this new attack surface, we identify critical gaps in current defences and propose a research agenda that could secure the entire agent ecosystem.
Cloudflare and OpenAI: making AI discovery better for everyone
How Cloudflare and OpenAI are working together to help AI find useful content, keep up with changes, and make fewer wasted requests – making the experience better for users and website owners alike.
