Google and Microsoft have joined OpenAI and Anthropic in supporting a shared robots.txt rule that lets online creators opt out of model training. The plan introduces a single line intended to replace the messy web of crawler rules webmasters run today. This new AI training opt-out tells AI labs not to feed page text into future models, but it does not block web spiders from visiting sites or undo past data collection [1].
The AI Training Opt-Out Mechanism
The plan comes from an Internet-Draft written by Anthropic standards lead Kevin Kelley and filed with the Internet Engineering Task Force (IETF). Site owners place a User-Agent line in robots.txt using the token __usage_ai-training to stop scrapers from taking content. The IETF AI Preferences (AIPREF) group defines training as tuning the learned weights of an AI model, excluding basic classification or search ranking tools. Under Kelley’s proposal, adding a standard disallow command lets webmasters block GPTBot, ClaudeBot, and Google-Extended in one move, cutting out the tedious chore of writing and maintaining separate configuration lines for each firm. This single signal gives website administrators a cleaner way to express content rules across multiple generative AI providers [2].
Site owners can grant partial access. A newsroom can block AI scrapers from articles and add an allow line for its press kit. This choice keeps media kits open to bots but shields news stories from model intake. The rule serves as a clear tag of intent [5].
Yet Kelley’s draft has no official IETF status and expires on May 3, 2027 unless the working group renews it. OpenAI said it plans to honor the signal for foundational generative AI training, a phrase that TechRadar reporter Rahim Amir noted sounds narrower than the AIPREF draft. Adopting an AI training opt-out changes how AI firms use data, not how bots fetch pages [1].
Search Visibility and Grounding Differences
Web creators worry that blocking AI training bots will hurt their rank on search engines. Google addressed that fear by separating Googlebot from Google-Extended. Googlebot crawls web pages to build Google Search results. Google-Extended gathers data for Gemini models. Maria Deutscher reported in SiliconANGLE that blocking training with the new token will not stop Googlebot from indexing normal web pages. But Google also explained on Search Central that if a site posts the shared token without naming Google-Extended, systems treat the rule as a block against Google-Extended [3].
That link creates a dilemma for online writers. Google-Extended controls model training and live grounding for Gemini answers. If an outlet blocks the token, it stops model training, but it may also lose Gemini citations, answer links, and search traffic because Google provides no tool to split those tasks [1].
Microsoft chose a different path by setting up a page-level control for AI grounding in Copilot. Krishna Madhavan, principal product manager for Microsoft Bing, said site owners “should not have to choose between being discoverable in Bing and deciding how generative models use their work” [4]. Can granular rules give webmasters real peace of mind? Microsoft lets webmasters add a DisallowExtended tag to keep content inside Copilot answers but block grounding data sold to outside businesses [1].
Past Scrapes Remain Outside the Rule
The new opt-out works only for future web crawls [5]. When a webmaster adds the tag today, AI labs will not wipe past text from stored files or excise it from completed checkpoints. Web crawlers run by OpenAI, Google, and Anthropic pulled millions of web pages over years of quiet data intake. Those articles, books, and forum posts stay locked inside current weights [1].
The tag brings no money. Microsoft stated plainly that these crawler choices do not establish licensing contracts or royalty systems, meaning publishers who seek compensation must negotiate private deals outside web protocols [4]. Commercial alliances continue to shift across the market, as shown when Replit picked OpenAI over Anthropic to power its coding agent.
Robots.txt operates as an honor system built on trust rather than digital locks. Spiders can easily ignore the tag. Web teams wanting ironclad safety against scrapers still turn to paywalls, login gates, and Cloudflare rate limits. These defensive barriers stop crawlers at the server gate [1].
Publisher Lawsuits and Copyright Disputes
Court fights pushed AI labs toward this common ground. In December 2023, The New York Times sued OpenAI and Microsoft, alleging that unauthorized scraping of journalistic archives infringed on protected copyrights. Both tech giants defended their scraping under fair use in United States courts. Anthropic agreed to pay $1.5 billion to settle copyright claims brought by authors over pirated books in training sets. In 2025, Penske Media, owner of Rolling Stone, sued Google over AI Overviews answer cards [1].
Litigation files also exposed sharp splits inside tech labs. Unsealed filings in the Times suit show that Microsoft director Brent Hecht wrote in a 2023 internal memo that AI data scraping was possibly “the largest theft of labor in human history”. That warning stood in stark contrast to words from Microsoft AI head Mustafa Suleyman in 2024. Suleyman called open-web content freeware for model training unless sites explicitly objected. The new token creates the exact objection tool Suleyman mentioned [1].
European rules also speed up adoption across the Atlantic. Google noted that this token matches European Union AI rules that recognize robots.txt as a primary way for websites to register machine-readable objections. Andy Stone, vice president of communications at Meta Platforms, called the joint token an important step for the web. Yet Meta has not agreed to obey the line. That hesitation followed intense public backlash when Meta revised its website crawling terms to drop AI training after commercial partners objected to sweeping terms [4]. Sam Quigley of the News/Media Alliance told the Mercury News that training still seems potentially on the table, keeping publishers cautious about future corporate policies [1].
AI Training Opt-Out Rollout Schedules
Participating AI labs will roll out the signal on separate dates. Anthropic already honors Kelley’s token in its live systems. Microsoft plans to recognize the rule as its Bing spiders scan the web starting February 1, 2027. Google aims to support the system in 2027, and OpenAI promised to adopt the tag in coming months without setting a firm calendar date [2].
Because dates differ, webmasters must keep old crawler blocks in place [5]. Gio Farley reported in Itech Post that dropping tags for GPTBot, ClaudeBot, or Google-Extended now leaves pages open to scraping before systems update their filters. Will one shared tag solve scraping fights across the internet? Probably not on its own. Cloudflare began blocking AI bots by default for new customers in July 2025 and lets site owners bill crawlers for page visits [1].
Tensions between web networks and AI labs remain high. In 2025, Cloudflare accused AI search startup Perplexity of faking bot identities to crawl blocked sites, an accusation Perplexity denied. The new AI training opt-out gives web creators a cleaner voice in robots.txt, but its success depends on whether tech firms honor the rules they helped write [1].
- ONLINE NEWS Amir, R. (2026, November 8). Google and Microsoft among those signing up to OpenAI’s plan for a new website crawling standard — including opting-out of internet scraping. TechRadar. [Article Link]
- ONLINE NEWS Moon, M. (2026, October 31). Google joins OpenAI and Anthropic in promising new scraping controls. Engadget. [Article Link]
- ONLINE NEWS Deutscher, M. (2026, October 30). Google, Anthropic and others to support new AI crawling opt-out standard. SiliconANGLE. [Article Link]
- ONLINE NEWS Navi. (2026, October 31). AI Scraping Opt-Out Standard Backed by Google, OpenAI. TheOutpost.ai. [Article Link]
- ONLINE NEWS Farley, G. (2026, November 3). Google, OpenAI and Anthropic Unite to Introduce New AI Scraping Controls. Itech Post. [Article Link]
- ONLINE NEWS Amir, R. (2026, November 8). Google and Microsoft among those signing up to OpenAI’s plan for a new website crawling standard — including opting-out of internet scraping. Yahoo Tech. [Article Link]
