







This document describes how a service operator binds usage preferences (e.g., "no AI training") to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User- Agent line. It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed.
Machine-Readable Training Preference Signal
This document describes how a service operator binds usage preferences (e.g., "no AI training") to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User- Agent line. It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed.
Making it easier for websites to express AI training preferences
OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers.

Supporting a shared AI training control for websites | Google Search Central Blog | Google for Developers
Today, we're announcing our plans to support a shared AI training control in robots.txt. This will help websites express their AI training preferences across multiple AI providers in a single step and is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF). This builds on our long history of providing website owners with clear controls, including through Google-Extended, the AI training control we've offered since 2023.

Content Signals
An up-to-date guide to the IETF's proposed new AI Preferences (aipref): a new way for website publishers to control how automated systems use their content.
DataLicenses.org
Machine-readable hints for AI agents/crawlers; easy to adopt, rely on compliance.
Meta drops the word "train" from its commercial TOS after AI backlash
Meta changes AI scraping policy but won’t add language respecting robots.txt.

Microsoft offers devs a better way to control AI agent behavior | TechCrunch
The specification lets developer, compliance, and security teams define their own policies for agents to follow in portable policy files.

Add community.lexicon.preference.ai lexicon by ngerakines · Pull Request #72 · lexicon-community/lexicon
Summary Introduces the community.lexicon.preference.ai lexicon for declaring user preferences regarding AI usage of their public data Decomposes AI usage into four distinct categories (training, i...
Permission Manifests for Web Agents
The rise of Large Language Model (LLM)-based web agents represents a significant shift in automated interactions with the web. Unlike traditional crawlers that follow simple conventions, such as robots$.$txt, modern agents engage with websites in sophisticated ways: navigating complex interfaces, extracting structured information, and completing end-to-end tasks. Existing governance mechanisms were not designed for these capabilities. Without a way to specify what interactions are and are not allowed, website owners increasingly rely on blanket blocking and CAPTCHAs, which undermine beneficial applications such as efficient automation, convenient use of e-commerce services, and accessibility tools. We introduce agent-permissions$.$json, a robots$.$txt-style lightweight manifest where websites specify allowed interactions, complemented by API references where available. This framework provides a low-friction coordination mechanism: website owners only need to write a simple JSON file, while agents can easily parse and automatically implement the manifest's provisions. Website owners can then focus on blocking non-compliant agents, rather than agents as a whole. By extending the spirit of robots$.$txt to the era of LLM-mediated interaction, and complementing data use initiatives such as AIPref, the manifest establishes a compliance framework that enables beneficial agent interactions while respecting site owners' preferences.

A Survey on Large Language Model based Autonomous Agents
Autonomous agents have long been a prominent research focus in both academic and industry communities. Previous research in this field often focuses on training agents with limited knowledge within isolated environments, which diverges significantly from human learning processes, and thus makes the agents hard to achieve human-like decisions. Recently, through the acquisition of vast amounts of web knowledge, large language models (LLMs) have demonstrated remarkable potential in achieving human-level intelligence. This has sparked an upsurge in studies investigating LLM-based autonomous agents. In this paper, we present a comprehensive survey of these studies, delivering a systematic review of the field of LLM-based autonomous agents from a holistic perspective. More specifically, we first discuss the construction of LLM-based autonomous agents, for which we propose a unified framework that encompasses a majority of the previous work. Then, we present a comprehensive overview of the diverse applications of LLM-based autonomous agents in the fields of social science, natural science, and engineering. Finally, we delve into the evaluation strategies commonly used for LLM-based autonomous agents. Based on the previous studies, we also present several challenges and future directions in this field. To keep track of this field and continuously update our survey, we maintain a repository of relevant references at https://github.com/Paitesanshi/LLM-Agent-Survey.

Curated skills for AI agents · skilld
A curated registry of agent skills for the npm packages and GitHub repos you actually use. One install command, every agent. Get notified when they change.

Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First
Large Language Model (LLM) agents, acting on their users' behalf to manipulate and analyze data, are likely to become the dominant workload for data systems in the future. When working with data,...

How AI Agents Are Changing API Architecture | daily.dev
Autonomous AI agents are reshaping API consumption patterns, prompting API providers to rethink identity, authorization, security, design, and observability....

New blog post: Signaling AI Preferences on ATProto. Introducing community.lexicon.preference.ai, a lexicon for granular AI data usage preferences with scoped overrides for specific entities and collections. #AI #atproto
Signaling AI Preferences on ATProto
ngerakines.leaflet.pubAs promised, we’ve created a policy document outlining our thoughts on AI and agentic coding (AI for software development). We’re releasing a vote later this week for Blacksky community members to offer their feedback. We look forward to hearing from you all.
Blacksky Algorithms' Policy Towards Agentic Coding
blackskyweb.xyz