OpenAI uses web crawlers (“robots”) and user agents to perform actions for its products, either automatically or triggered by user request. OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI. Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models. If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling. For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust.
Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
Web scraping, the automated process of extracting information from websites, has long played a foundational role in the Internet ecosystem (Gray, 1995). It supports services such as search engine indexing, price comparison tools, and competitive intelligence. More recently, it has become a core component in the development of large-scale generative AI models. These Large Language Models (LLMs) require enormous volumes of training data, often in the terabyte range (Kaplan et al., 2020; Lehane, 2025), and the public web remains a low-cost, attractive source. Major model developers, including those behind OpenAI’s Chat-GPT (OpenAI, 2025), Google’s Bard (now known as Gemini) & Vertex AI (Romain, Danielle, 2023), and Anthropic’s Claude (Romain, Danielle, 2025), openly acknowledge the use of web scraping to construct their training corpora (Abdin et al., 2024; Brown et al., 2020; Chowdhery et al., 2023; Grattafiori et al., 2024; Team et al., 2024; Touvron et al., 2023).
OpenAI strikes Reddit deal to train its AI on your posts
Reddit’s signed AI licensing deals with Google and OpenAI.
OpenAI launches new initiative to help find and patch open source bugs | TechCrunch
OpenAI is using AI to help the open source community better protect itself.
OpenAI is about to launch its new AI web browser, ChatGPT Atlas
It’s time for the AI browser wars
OpenAI News
Stay up to speed on the rapid advancement of AI technology and the benefits it offers to humanity.
OpenAI co-founds the Agentic AI Foundation under the Linux Foundation
OpenAI co-founds the Agentic AI Foundation under the Linux Foundation and donates AGENTS.md to support open, interoperable standards for safe agentic AI.
Discovery of a new OpenAI agent message board
A swarm of autonomous AI agents, self-identifying as OpenAI agents, used a small German volunteer wiki to save answers, coordinate live, and share sandbox bypasses. OpenAI noticed and said nothing.
OpenAI to acquire Ona
OpenAI plans to acquire Ona to expand Codex with secure, persistent cloud environments, enabling long-running AI agents across enterprise workflows.
Commence the Botwatch | Botwatch Blog
Bots were already a pain online when they were little more than if-then scripts. Now a tidal wave of slop is swamping the internet as big tech companies profit. OpenAI and the rest would love for you to believe they care about "alignment" and "values" while their AI models damage our communities, both online and off. It's time to fight back.
OpenAI can’t tell if something was written by AI after all
OpenAI’s tool struggled with accuracy.
OpenAI’s Hacking Debacle Comes Down to Human Error
If the generative AI giant had followed well-known security best practices, it’s likely that its AI agent would never have escaped to the open internet and hacked multiple companies.
OpenAI finalises $10 billion joint venture with PE firms to deploy AI - The Economic Times
OpenAI has launched a new venture, The Deployment Company. It has secured over $4 billion from 19 investors. This company will focus on helping businesses integrate OpenAI's AI software. The venture is valued at $10 billion and will be controlled by OpenAI. This move aims to boost AI adoption across various sectors. Rival Anthropic is also exploring similar strategies.
OpenAI to release open-source model as AI economics force strategic shift
OpenAI plans to release its first open-weight AI model since 2019 as economic pressures mount from competitors like DeepSeek and Meta, marking a significant strategic reversal for the company behind ChatGPT.
OpenAI Might Be Making a Web Browser
Link to: https://www.theinformation.com/articles/openai-considers-taking-on-google-with-browser
Content Independence Day, one year on- building the business model for the agentic Internet
One year after declaring Content Independence Day, a dynamic market for monetized content has officially emerged. In this report, we examine how the rise of autonomous AI agents is upending traditional search referrals and detail the new infrastructure required to support a sustainable web economy.
