Creepy crawlies — Konstantin Ryabitsev

DataLicenses.org

Machine-readable hints for AI agents/crawlers; easy to adopt, rely on compliance.

Trapping misbehaving bots in an AI Labyrinth

How Cloudflare uses generative AI to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.

Anti-Crawler Defenses | The Paper Pilot

The Paper Pilot's digital garden of thoughts on philosophy, politics, and sociology

2025 Index — AI Agent Index

Interactive browser for 30 AI agents across 45 annotation fields in the 2025 AI Agent Index.

Getting Bots to Respect Boundaries

Getting Bots to Respect Boundaries How AI Crawlers Are Straining Web Infrastructure Audrey HingleJanuary 2026 Image by Janet Turra & Cambridge Diversity Fundbetterimagesofai.org creativecommons.org/licenses/by/4.0 Contents Introduction: Setting up the Problem 3 Understandin...

"Build protocols, not platforms"

Exploring the 'Protocols, Not Platforms' paper alongside a curated digest of recent AI crawler policy shifts and developer experiments.

Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.

Agency in the Age of AI

A new special issue and the Robot Boogie

An update on the scraper situation

Our article 'Fighting the AI scraper bot scourge', published in early 2025, discussed the probl [...]

Ultimate Block List to Stop AI Bots | Perishable Press

More than you might think, AI (Artificial Intelligence) and ML (Machine Learning) bots are crawling your site and scraping your content. They are...

AI as a Tool for the Mental Load | Brittany Ellich | Offprint

AI didn't make me faster at tasks. It took over the tracking, the invisible remembering that runs a household, and gave me back creative energy I forgot I had

Is there something it is like to be an AI?

Posted on Wednesday 2 Jul 2025. 1,593 words, 6 links. By Matt Webb.

The 2025 AI Agent Index

Agentic AI systems are increasingly capable of performing complex tasks with limited human involvement. The 2025 AI Agent Index documents the origins, design, capabilities, ecosystem, and safety features of 30 prominent AI agents based on publicly available information and correspondence with developers.

Clawdbot — Personal AI Assistant

Clawdbot — The AI that actually does things. Your personal assistant on any platform.