OpenAI, Google, Microsoft and Anthropic Back One robots.txt Signal for AI Training Opt-Outs

OpenAI, Google, Microsoft and Anthropic said on October 30, 2026 that they will honor one shared signal in robots.txt that tells AI developers whether a site's content may be used to train generative AI models. A site adds a single rule group for the policy token __usage_ai-training instead of writing separate rules for each company's crawler.

Until now, opting out of AI training meant naming crawlers one by one, such as OpenAI's GPTBot or Google's Google-Extended. The question of who controls that choice has been a recurring policy fight between publishers and AI companies, and the new token comes out of the Internet Engineering Task Force (IETF), the body that sets many core internet standards, where an AI Preferences working group has been defining the vocabulary. Search Engine Roundtable reported the announcements first, according to The Verge.

What did the four companies announce?

Each company published its own commitment on October 30, 2026. OpenAI said in its announcement that it will honor the common signal as a voluntary effort, aims to implement it "in the coming months," and encouraged other AI providers to honor it too.

Google said in a Search Central post by John Mueller, a Google Search Advocate, that it aims to support the control in 2027. Microsoft said in a Bing blog post that it will begin honoring the new preferences as it crawls on February 1, 2027. For Anthropic, the commitment came in a message to the IETF working group's mailing list from Kevin Kelley, the draft's author, who wrote that Anthropic "has already started implementing support", adding that rollout across its services will take time.

All four say the shared token sits alongside their existing controls rather than replacing them. OpenAI says its GPTBot and other crawler rules will continue to apply, and Google says Google-Extended, its AI training control since 2023, keeps working.

How does the __usage_ai-training token work?

The token is an ordinary robots.txt rule group whose User-Agent line names a purpose instead of a crawler, according to the IETF draft published on October 30, 2026. A Disallow line for a path expresses a preference against AI training there, and an Allow line expresses a preference in favor.

The draft's own example, a group for __usage_ai-training with Disallow: / and Allow: /press-kit/, opts a whole site out of training except its press kit. Parsers that do not know the token ignore it. Three design choices stand out:

• Wildcards do not count. A general User-Agent: * group is never read as matching the token, so a robots.txt file without a __usage_ai-training group expresses no training preference at all, Kelley wrote.
• Use, not access. The draft says the token "is not a directive for crawling behavior." A company may skip fetching pages it could not use, but limiting crawling still takes rules for each crawler's own name.
• Narrow definition. AI training means using content to change the learned parameters of a generative AI model, a definition taken from the working group's vocabulary draft. Training models used only for non-generative tasks such as ranking is not covered.

How will each company apply it, and when?

The four companies have committed to the same token but not to the same date or the same mapping onto their own systems, according to their posts. Google says that when a site opts out through the token and does not mention Google-Extended, it will treat that as if the site had disallowed Google-Extended.

Microsoft goes further than the shared signal. It is adding a page-level training preference that can override robots.txt, and a separate AI Grounding preference for whether pages may inform AI answers in Bing, Copilot and Microsoft 365. A new DisallowExtended option keeps content in Microsoft's own products but out of grounding services it sells to third parties. The new settings replace the NoArchive and NoCache tags Microsoft has offered since 2023, which will no longer count as training or grounding preferences after March 1, 2027.

Who is not on the list yet?

The companies named in reports by The Verge and Search Engine Roundtable on October 30, 2026 are OpenAI, Google, Microsoft and Anthropic, and the list does not include Meta. That stands out because two Meta employees appear in the draft's acknowledgments among people who shaped the working group's discussion (the draft lists affiliations for identification only).

Meta faced its own robots.txt dispute two weeks earlier. On October 16, 2026, it removed the word "train" from a crawling clause in its new business terms after publishers objected, while promising in statements, not in the contract, that robots.txt preferences still apply, as covered in the report on Meta's crawler terms.

The draft itself is also early. It is an individual Internet-Draft, which the IETF says "has no formal standing" in its standards process, and it expires on May 3, 2027 unless revised. Google says the approach is consistent with the EU's Code of Practice for general purpose AI model providers, which treats robots.txt as a primary way for publishers to state machine-readable preferences.

What we don't know yet

• Whether Meta, Amazon, Apple or other AI developers will honor the token.
• When Anthropic's support will cover all of its services, and how OpenAI will implement it; OpenAI says it will share details later.
• Whether the IETF working group will adopt the draft, and whether the token changes on the way to a final standard.
• How any of the four companies will show, or let outsiders check, that they honor the preference.