OpenAI, Google, Microsoft, and Anthropic have declared their intent to adopt a common standard for opting out of AI crawling, potentially making the rules governing data use between publishers and AI companies more concrete. Publishers have noted that a “policy token,” which can manage AI training preferences across multiple companies in one place, is expected to serve as a unified signal. The effectiveness of AI crawling opt-outs depends not only on the common declaration but also on how strongly each provider enforces it.
A Common Signal Inserted Into robots.txt
The proposed policy token would be placed inside a website’s robots.txt file. Until now, robots.txt has served as a place to indicate the scope of access permitted to web crawlers, including search indexing, archiving, and AI training. Rather than repeatedly communicating a choice to allow or restrict AI training for each company, the key advantage is that a single common signal can notify multiple providers of a site’s preferences.
Microsoft offers granular controls that allow users to opt out of AI training and AI grounding separately, so its approach should not be understood in exactly the same way. AI grounding is the process of connecting an AI model to verified information sources to reduce hallucinations. Regardless of Microsoft’s controls, websites will continue to appear in Bing search results. Google has also introduced similar controls that separate whether content appears in AI search features from whether it is used for AI training.
The Gap Between Declarations and Actual Blocking
A common policy token is an agreed-upon tool, but it does not become a wall on its own. OpenAI, Google, Microsoft, and Anthropic have said they will restrict AI crawler access when they find a policy token on a site. The actual level of blocking and the degree of policy granularity remain areas for providers to determine, and adopting the standard by supporting a unified signal does not guarantee equivalent enforcement.
The industry’s response has already split into several different approaches. Really Simple Licensing provides a standard that can require visiting crawlers to pay, while Cloudflare blocks AI crawlers by default. Rather than converging the choices among usage restrictions, payment, and default blocking, policy tokens are closer to serving as a common linking signal for multiple control methods.
Support Timelines and Enforcement Responsibility
Implementation schedules differ among providers. Anthropic has already introduced the new signal, while Google and Microsoft plan to support it next year. OpenAI has said it will implement the standard within the coming months. Meta communications head Andy Stone has expressed support but has not said whether the company will actually follow the policy token.
Andy Stone said a common signal that communicates preferences to multiple providers at once is an important step. He said he expects to establish the standard with other AI companies and see it implemented across the industry. This shared objective does not define the operational behavior of the common signal, how exceptions will be handled, or how policy changes will be propagated.
The next stage of AI crawling opt-outs depends more on enforcement responsibility than on the token itself. Microsoft’s distinction between granular controls and search exposure policies, Google’s separate management of AI search features and training, and Anthropic and Microsoft’s plans to introduce the signal next year show that provider-specific interpretations will remain even after a unified signal. Publishers can now express common preferences, but the actual blocking effect can only be confirmed by watching how each company applies the rules.
Source
This article was written based on the following source: The Verge — The biggest names in AI are backing a new way to opt out of scraping

Leave a Reply