Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
Content Independence Day: no AI crawl without compensation!
It’s Content Independence Day: Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content.
Patreon Blocks Crawlers From Stealing Creators' Work for AI Training
“Creators deserve credit, compensation, and consent. If that's not on the table, the crawlers can stay the fuck off Patreon," CEO Jack Conte wrote on Thursday.
Cloudflare's Matthew Prince has a plan to get Google and the AI oliigarchs to pay for your content even though many are used to getting it for free. He might have enough leverage to make them.
AI chatbots are blowing up the 30-year economic relationship publishers have had with search engines. Many think this will kill the web if not addressed. Google needs to change first. It's resisting. Cloudflare's Matthew Prince is going to try and make them.
Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf, and allowing targeted blocking of AI crawling on sites that serve ads.
Content Independence Day, one year on- building the business model for the agentic Internet
One year after declaring Content Independence Day, a dynamic market for monetized content has officially emerged. In this report, we examine how the rise of autonomous AI agents is upending traditional search referrals and detail the new infrastructure required to support a sustainable web economy.
DataLicenses.org
Machine-readable hints for AI agents/crawlers; easy to adopt, rely on compliance.
"Build protocols, not platforms"
Exploring the 'Protocols, Not Platforms' paper alongside a curated digest of recent AI crawler policy shifts and developer experiments.
Nobody Wants to Pay for Your AI
AI is everywhere today. It writes social posts, suggests email replies, recommends your next binge-watch, and quietly powers tools you use without even noticing.
Exclusive: Multiple AI companies bypassing web standard to scrape publisher sites, licensing firm says
Multiple artificial intelligence companies are circumventing a common web standard used by publishers to block the scraping of their content for use in generative AI systems, content licensing startup TollBit has told publishers.
Getting Bots to Respect Boundaries
Getting Bots to Respect Boundaries How AI Crawlers Are Straining Web Infrastructure Audrey Hingle January 2026 Image by Janet Turra & Cambridge Diversity Fund betterimagesofai.org creativecommons.org/licenses/by/4.0 Contents Introduction: Setting up the Problem 3 Understandin...
How AI bots quietly dismantle paywalls via web search
ChatGPT and other AI chatbots have figured out how to get around paywalls through "live" web search—and they're doing it systematically and quietly across major publications, new Digital Digging research reveals.
The Technical Feasibility of Divesting Google Chrome – Knight-Georgetown Institute
As the European Commission advances efforts under the Digital Markets Act to require Google to share its search data with competitors, lessons from historic antitrust remedies underscore how data access could be transformational in the AI-powered search market. While the Commission’s proposals represent a novel and comprehensive approach, key improvements to data scope and sharing frequency, privacy protections, and dispute resolution are needed. US courts and enforcers charged with implementing similar provisions should take note.
Data Streaming for AI: From Extractive Training to Sovereign Infrastructure DWeb Camp 2026
AI systems are consuming the world's content without compensating its creators. This session explores data streaming as a new paradigm — where content flows to AI in real time, with built-in rights management, usage tracking, and fair compensation — and asks what it would take to make this infrastructure decentralized, sovereign, and governed by the communities it serves.
The Agentic Web and Original Sin
Microsoft is putting forth compelling proposals for the Open Agentic Web. However, the proposal needs digital payments, which will be key to creating a new content marketplace for AI.
Mistral CEO: AI companies should pay a content levy in Europe
A revenue-based charge would protect the livelihoods of copyright holders and bring legal certainty
Google’s broken link to the web
With AI search results coming to the masses, the human-powered web recedes further into the background
