Nowadays, much of the internet is explored not by humans, but by machines. Invisible programs known as crawlers move through websites, following links, reading pages, and collecting information. They are fundamental to how the internet is indexed, organized, and increasingly used as a source of material for artificial intelligence. Yet, despite their constant activity, we rarely encounter these machines or consider their behavior. In Crawlers, Slava gives these invisible processes a body, a voice, and a habitat, transforming the automated extraction of information into a living, playful, and occasionally chaotic experience.
Common Crawl - Open Repository of Web Crawl Data
We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.
Introducing SlopStop: Community-driven AI slop detection in Kagi Search
Your collective defense against AI-generated spam and content farms We made it our mission to prevent the web from becoming useless and a harmful space. That’s why today, Kagi Search introduces the first community-driven system to...
DataLicenses.org
Machine-readable hints for AI agents/crawlers; easy to adopt, rely on compliance.
"Build protocols, not platforms"
Exploring the 'Protocols, Not Platforms' paper alongside a curated digest of recent AI crawler policy shifts and developer experiments.
Semble: Collective knowledge foraging in trail-based social networks beyond feeds (Lightning Talk)
Creepy crawlies — Konstantin Ryabitsev
You've probably heard me complain about the "AI crawlers" before, but now I actually have some hard numbers I can put up to show their im...
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
AI code and software craft - alex wennerberg
Much has been said about audio, video and text "slop": low-quality, AI-generated content that has proliferated on the internet since the release of publicly-accessible AI models. Garbage content has always existed online, but the novelty of AI is that it has made its generation orders of magnitude less labor-intensive. For anyone who lacks a discerning eye, or is doing some task where discernment simply does not matter, AI has become a sufficient replacement for human hands.
AI-generated ‘slop’ is slowly killing the internet, so why is nobody trying to stop it? | Arwa Mahdawi
Low-quality ‘slop’ generated by AI is crowding out genuine humans across the internet, but instead of regulating it, platforms such as Facebook are positively encouraging it. Where does this end, asks Arwa Mahdawi
Anti-Crawler Defenses | The Paper Pilot
The Paper Pilot's digital garden of thoughts on philosophy, politics, and sociology
We're making it easier to identify AI-generated content globally.
We’re launching a standalone platform to help you easily identify whether online content was created using Google AI or tools from our industry partners.
Keynote: The Death of the Browser - Rachel-Lee Nabors, AgentQL
AI Slop is Killing Online Communities
Like a young child coming home from kindergarten with their latest crayon scrawls, the internet is currently awash with people sharing their AI-generated work. And just like the young child’s drawings, much of that work should be proudly put up on the walls within the artist’s house—and no further.
A deep dive into OpenSEO
Explore OpenSEO's architecture: SEO data APIs, Cloudflare Workers, its audit crawler, MCP tools, AI agents, storage, and self-hosting trade-offs.
Google’s broken link to the web
With AI search results coming to the masses, the human-powered web recedes further into the background
