Also, all other considerations and opinions on using large language models, coding and document creation agents, ... left aside for a moment: Why spend tokens, effort, storage, bandwidth to populate web pages with content "created" by automated processing of resources publicly online? If we advocate and support use of these tools for this particular purpose, wouldn't it be safe to assume people interested in this information and accepting use of these tools will be easily ready and able to do this kind of processing in a way digestable to them individually on their own?
The Impact of AI-Generated Text on the Internet
By mid-2025, roughly 35% of newly published websites were classified as AI-generated or AI-assisted.
The Mediated Web
<p>Large Language Models have the potential to hugely shift what it means to interact with a web browser.</p> <p>Here’s a prediction: the web will soon be mediated by agents, not just rendered by web browsers. Every user will have their own AI interpreting information for them, and executing tasks on their behalf.</p> <p>What does this mean now, and what does it mean for the future of the web? Let’s delve into what a mediated web might look like, and how it could impact the internet ecosystem.</p>
Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
Web scraping, the automated process of extracting information from websites, has long played a foundational role in the Internet ecosystem (Gray, 1995). It supports services such as search engine indexing, price comparison tools, and competitive intelligence. More recently, it has become a core component in the development of large-scale generative AI models. These Large Language Models (LLMs) require enormous volumes of training data, often in the terabyte range (Kaplan et al., 2020; Lehane, 2025), and the public web remains a low-cost, attractive source. Major model developers, including those behind OpenAI’s Chat-GPT (OpenAI, 2025), Google’s Bard (now known as Gemini) & Vertex AI (Romain, Danielle, 2023), and Anthropic’s Claude (Romain, Danielle, 2025), openly acknowledge the use of web scraping to construct their training corpora (Abdin et al., 2024; Brown et al., 2020; Chowdhery et al., 2023; Grattafiori et al., 2024; Team et al., 2024; Touvron et al., 2023).
Who makes money when AI reads the internet for us? - Engadget
Instead of displaying links, Arc Search's "Browse for Me" feature reads the first handful of pages and summarizes them into a single, custom-built, Arc-formatted web page using large language models from OpenAI and others. Critics say that's a problem.
We're making it easier to identify AI-generated content globally.
We’re launching a standalone platform to help you easily identify whether online content was created using Google AI or tools from our industry partners.
Content Independence Day, one year on- building the business model for the agentic Internet
One year after declaring Content Independence Day, a dynamic market for monetized content has officially emerged. In this report, we examine how the rise of autonomous AI agents is upending traditional search referrals and detail the new infrastructure required to support a sustainable web economy.
The Web’s Creator Looks to Reinvent It (Published 2016)
Tim Berners-Lee and other computer scientists are pondering newer technologies to create a web with more privacy and less government control.
Introducing Markdown for Agents
The way content is discovered online is shifting, from traditional search engines to AI agents that need structured data from a Web built for humans. It’s time to consider not just human visitors, but start to treat agents as first-class citizens. Markdown for Agents automatically converts any HTML page requested from our network to markdown.
Opinion | How to Save the Web (Published 2018)
Governments, businesses and users around the world must commit to a set of fundamental principles for the web.
Curated retrieval versus open web search in public AI information...
Public institutions increasingly use large language models (LLMs) to answer citizens' questions, often pairing a curated knowledge base with live web search, yet whether the sources behind these...
The 49MB Web Page
A look at modern news websites. How programmatic ad-tech, huge payloads and hostile architecture destroyed the reading experience.
Tumblr and WordPress to Sell Users’ Data to Train AI Tools
Internal documents obtained by 404 Media show that Tumblr staff compiled users' data as part of a deal with Midjourney and OpenAI.
The Consent Layer: Using ligatures to make web text expensive to scrape without asking
Publish for humans, not for crawlers. A web font that swaps the words in your HTML, so readers see your writing and AI training gets a stale copy.
The Better Web / The Human Web / The Living Web
A response to Aaron Parecki's response: on the false dichotomy of rewriting with Claude or shutting down, why 'understanding matters more than tools' sever…
