Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf, and allowing targeted blocking of AI crawling on sites that serve ads.
Have it both ways: stay discoverable in search while disallowing AI training
Cloudflare is giving site owners a way to stay discoverable while disallowing AI training. New controls and an Accountable designation establish a shared model with Apple, Google, and Microsoft.
Content Independence Day: no AI crawl without compensation!
It’s Content Independence Day: Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content.
Cloudflare and OpenAI: making AI discovery better for everyone
How Cloudflare and OpenAI are working together to help AI find useful content, keep up with changes, and make fewer wasted requests – making the experience better for users and website owners alike.
Trapping misbehaving bots in an AI Labyrinth
How Cloudflare uses generative AI to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.
Ultimate Block List to Stop AI Bots | Perishable Press
More than you might think, AI (Artificial Intelligence) and ML (Machine Learning) bots are crawling your site and scraping your content. They are...
Cloudflare gives AI agents an identity and a wallet | Cloudflare
New tools mean businesses will be able to see who's behind an AI agent, and those agents will be able to pay for things safely on their owner's behalf.
Cloudflare's Matthew Prince has a plan to get Google and the AI oliigarchs to pay for your content even though many are used to getting it for free. He might have enough leverage to make them.
AI chatbots are blowing up the 30-year economic relationship publishers have had with search engines. Many think this will kill the web if not addressed. Google needs to change first. It's resisting. Cloudflare's Matthew Prince is going to try and make them.
The age of agents: cryptographically recognizing agent traffic
Cloudflare now lets websites and bot creators use Web Bot Auth to segment agents from verified bots, making it easier for customers to allow or disallow the many types of user and partner directed bots the explosion of AI agents has created.
A Strategy for Protecting Content from AI
Because that’s the world we live in these days.
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
Meta seeks to scrape news content for AI training without permission | Angela Mills Wade posted on the topic | LinkedIn
Meta’s message to publishers: let us scrape your journalism for AI, or stop using our platforms. In the very week that News Media Coalition launched their Primary Source Journalism Framework we get another extraordinary example of how dominant digital platforms are attempting to turn their market power into permission to appropriate the content of others. For free. As reported by Nieman Journalism Lab, Meta’s new commercial terms, effective 30 October, purport to authorise the company to scrape websites associated with business accounts, including news publishers, and use that content to train its products. No negotiated licence. No meaningful choice. Just acceptance by continued use of Meta’s services. And the implications go further. The terms even contemplate creating test credentials to access websites, raising serious questions about publishers’ ability to control access to their own content. Meta argues that publishers can still use robots.txt to restrict crawling but this is no safeguard to protect primary source journalism as such sweeping ‘contractual’ permissions could undermine publishers rights reservations. This is becoming a familiar pattern. Google effectively ties visibility in Search to the use of publishers’ content in its AI services. Now Meta appears to be seeking similarly expansive rights through its commercial terms. This is not consent freely given. It is an attempt to leverage platform dependency to extract valuable intellectual property without negotiation or remuneration. European copyright law gives publishers rights over their content, including the right to reserve it from text and data mining. Those rights must not be circumvented through take-it-or-leave-it contractual conditions imposed by powerful gatekeepers. The European Commission and competition authorities should scrutinise these terms urgently, including their implications for copyright, contractual fairness and abuse of market power. Publishers must retain the right to decide whether, how and on what terms their journalism is used by AI companies. Opt-in, not opt-out. Licensing, not appropriation. Henna Virkkunen Teresa Ribera #Copyright #AI #PrimarySourceJournalism #MediaPluralism #Competition #Publishers #BadBots | 23 comments on LinkedIn
Data Streaming for AI: From Extractive Training to Sovereign Infrastructure DWeb Camp 2026
AI systems are consuming the world's content without compensating its creators. This session explores data streaming as a new paradigm — where content flows to AI in real time, with built-in rights management, usage tracking, and fair compensation — and asks what it would take to make this infrastructure decentralized, sovereign, and governed by the communities it serves.
Content Signals
An up-to-date guide to the IETF's proposed new AI Preferences (aipref): a new way for website publishers to control how automated systems use their content.
Cloudflare's role in web content control #aiengineering #webdevelopment
Exclusive: Multiple AI companies bypassing web standard to scrape publisher sites, licensing firm says
Multiple artificial intelligence companies are circumventing a common web standard used by publishers to block the scraping of their content for use in generative AI systems, content licensing startup TollBit has told publishers.
Introducing Kitesurf: The agent-first browser that runs in V8 isolates on Cloudflare Workers
We should be giving all agents tools that excel at what’s important for an AI model. Kitesurf is Cloudflare’s new stateless, highly scalable, and cost-effective web browser that runs entirely on top of Workers and was designed specifically for the Agentic Cloud.
