AI Training Opt-Out 2026: One robots.txt Rule Explained

For years, telling AI companies not to train on your website meant learning a new rule for every company. Google had Google-Extended. OpenAI had GPTBot. Anthropic had ClaudeBot. Each switch worked differently, covered different things, and had to be maintained separately. On 30 October 2026, the four biggest Western AI labs — OpenAI, Google, Microsoft and Anthropic — said they would read one line the same way.

That line is a shared robots.txt token: __usage_ai-training. It is the most useful simplification publishers have had in the AI training fight — and it is also easy to over-read. It does not take effect everywhere at once, it does not do everything people assume it does, and it is not yet a finished standard. Here is what it actually is, when each company starts honouring it, and what to do with it today.

The two lines at the heart of it

The token lives in robots.txt, the plain-text file at the root of a website that tells automated crawlers what they may fetch. A site that wants to opt its whole site out of AI training adds this block:

User-Agent: __usage_ai-training
Disallow: /

That is it. The token goes where a crawler name normally goes, and Disallow: / applies the preference to every path. Sites that want a partial opt-out can scope it instead — for example, disallowing training on everything except a press kit by adding an Allow: line for that path, or disallowing only one folder.

Two things about the format matter. First, the token must match exactly: it is __usage_ai-training, with two leading underscores. A near-miss spelling is just an unknown user-agent that no crawler will act on. Second, this is a policy token, not a crawler name. No bot is actually called __usage_ai-training; the four labs have agreed to treat this token as a shared instruction about how content may be used, rather than as a rule about which bot may visit.

The proposal comes from an Internet-Draft titled "Machine-Readable Training Preference Signal", written by Kevin Kelley of Anthropic and dated 30 October 2026, in the Internet Engineering Task Force's AI Preferences working group. That status needs stating plainly: an individual Internet-Draft is an early-stage proposal, not a finished standard and not endorsed by the IETF itself. It expires on 3 May 2027 unless it is revised. The labs are backing a convention while the formal standard catches up — which is why their individual rollout dates, not the announcement date, are what actually matter.

Why this took so long to agree

The mess the token replaces was not an accident. Each lab built its controls at a different moment, for a different product, under different legal pressure. Google-Extended, introduced in 2023, was a usage control bolted onto robots.txt. OpenAI's GPTBot and Anthropic's ClaudeBot were crawler names with their own rules. A publisher with a real opt-out policy had to track all of them, and a mistake in any one left a gap. The shared token collapses that maintenance into one entry — but only for the companies that adopt it, and only for training.

When each lab actually starts

The announcement was simultaneous. The implementation is not. Here is the timetable as each company has stated it:

OpenAI's own announcement is the most careful of the four: it says the new signal will not replace the controls it already provides, and that its existing crawler controls will continue to apply and be respected. Google says the same about Google-Extended, with one detail worth knowing — if a site opts out using the shared token without explicitly mentioning Google-Extended, Google plans to treat that as if Google-Extended had been disallowed. An explicit Google-Extended rule, where one exists, overrides the shared token for Google.

The honest summary is this: nothing changes at most companies until 2027. A site that adds the token this weekend has expressed a preference that, for three of the four labs, will only be read next year. That is not a reason to dismiss it — getting the line in place before the rollout is exactly how a preference system is supposed to work — but anyone writing "add this line and you are opted out" today is overstating it.

The three things it does not do

1. It does not block crawling. The draft is explicit that the token is not a directive about crawling behaviour. A bot may still visit, fetch and cache a page; the token only states a preference about training use afterwards. Robots.txt has always worked on trust in this sense — it is a published preference, not a lock — and this token extends that trust-based model to a new question rather than replacing it with enforcement.

2. It does not cover search results or AI answers. Opting out of training does not remove a page from Google Search or Bing, and neither company says otherwise — both confirm search eligibility is unaffected. It also does not stop a chatbot from quoting a page in an answer. That is retrieval, not training: the model is not learning from your page, it is reading it live and summarising it. For publishers losing traffic to AI answers, that distinction is painful, because the use costing them visits is precisely the one this token does not touch. Our earlier look at Google's AI Mode links update explains why that traffic question is a separate fight.

3. It does not undo the past. Updating robots.txt does not remove content already collected into datasets, and it cannot un-train a model that has already been trained on your work. The token governs future training runs only. Publishers who think of it as a delete button will be disappointed; publishers who think of it as a forward-looking preference will use it correctly.

There is a fourth limit, less often stated: the token binds only the companies that choose to follow it. Smaller model builders, dataset collectors and scrapers that ignore robots.txt altogether are untouched by any announcement. A training opt-out is a signal to well-behaved crawlers, not a fence around your content.

Microsoft went further than the other three

Microsoft's announcement, published on the Bing blog on 30 October 2026, is the most detailed of the four — and the only one that addresses the AI-answers gap directly.

Alongside the shared robots.txt token, Microsoft is adding two page-level controls, set with meta tags or HTTP headers rather than in robots.txt. The first is a page-level training preference that overrides the site-wide robots.txt setting for an individual page, so a site can opt out globally but allow training on, say, its documentation pages. The second is a separate AI Grounding preference, which controls whether a page may be used to inform generated responses in Bing's AI experiences. Grounding has no site-wide robots.txt version — it is page-level only — and it comes in three settings: Allow (the default), Disallow, and DisallowExtended.

DisallowExtended is the interesting one. It limits grounding use to Microsoft's own experiences — AI Mode, AI Overview, Copilot and Microsoft 365 — while excluding the grounding services Microsoft sells to third parties. Microsoft is candid about the cost: choosing Disallow or DisallowExtended will significantly reduce a page's visibility in AI-generated experiences, though it does not affect eligibility for ordinary Bing search results. Training and grounding are independent choices, so a site can block training while allowing grounding, or the reverse.

Microsoft's new preferences also replace its old NoArchive and NoCache tags for training and grounding. Sites still relying on those tags have a hard deadline: they must move to the new preferences by 1 March 2027, after which the old tags will no longer be read as a training or grounding preference. A temporary grounding preference in Bing Webmaster Tools, opening 15 December 2026, bridges the gap for those sites, but there is no equivalent temporary training setting — another reason not to leave the migration to the last week.

The Meta story is a different fight

It is tempting to fold Meta into this announcement. Do not. Meta is not a party to the four-lab agreement, and its parallel story runs on an entirely different mechanism: contract terms, not robots.txt.

Meta's revised commercial terms for businesses — which publishers feared would let Meta crawl linked websites and use the content for its own products, including AI features — were originally due to take effect on 30 October 2026. After publisher backlash, including the removal of the word "train" from the clause, Meta pushed the date to 20 November, then paused the rollout indefinitely pending what its communications chief Andy Stone called "a full consultation". On 28 October, an Oslo court temporarily barred the terms after a challenge by ten Norwegian publishers. On 30 October, the UK's Competition and Markets Authority opened an investigation into whether the terms abuse a dominant position. Our earlier post on Meta's AI training clause removal covers how the clause evolved; the new developments are a court order and a regulator's investigation, not a technical opt-out.

The two stories share a theme — who controls web content in the AI era — but they are not the same mechanism, and confusing them leads to bad advice. The four-lab token is voluntary and forward-looking. Meta's terms are a contract fight about what businesses already agreed to by using Meta's products.

What to do today: a practical checklist

If you run a website and want to use the new token, the sensible sequence is:

1. Add the shared token now if you want the future coverage. Two lines in robots.txt, scoped with Allow: lines for any sections you are happy to share. It costs nothing and it will be read as each lab switches on.
2. Keep every existing control in place. Your GPTBot, Google-Extended and ClaudeBot rules are what actually work today. Deleting them in favour of a token that three labs have not switched on yet would leave you less protected, not more.
3. If you use Microsoft's NoArchive or NoCache tags, diary 1 March 2027. Move to the new training and grounding preferences well before then, and use the Webmaster Tools transition option for grounding if you need it.
4. Decide training and grounding separately, at least for Microsoft. Blocking training does not block AI answers, and blocking grounding has a visibility cost. Make each choice on its own merits rather than assuming one switch covers both.
5. Do not expect retroactive effect. If content already collected is your concern, robots.txt is the wrong tool — that is a licensing, legal or platform-complaint question, not a configuration change.

And if you are deciding whether to opt out at all, weigh the trade honestly. Opting out protects future training use of your work, which matters most for original reporting, distinctive writing and proprietary data. Staying in keeps your content eligible for the AI experiences that increasingly mediate discovery — and, for Microsoft grounding, visibly present in them. There is no universally right answer; there is only the answer that matches what your content is worth to you and where your readers actually find you.

Why this still matters despite the caveats

It is easy to read all of this and conclude the announcement changes nothing. That would be wrong too. For the first time, the four labs that matter most have agreed on the same language for the same question, in a document moving through a standards body, with dated commitments attached. The previous era — a dozen switches, each maintained separately — is ending, even if the new era arrives on four different dates.

Recommended Reading

If this debate has you thinking about how machine learning actually works — and why training data became the most contested asset on the web — Pedro Domingos' The Master Algorithm is a clear, non-technical tour of the quest for a single learning algorithm. Read more on The Master Algorithm at BookNest.

As an Amazon Associate, AI Trends Daily earns from qualifying purchases.

The featured image is an original AI-generated illustration, not a photograph. Sources: OpenAI, Google Search Central, Microsoft Bing blog, IETF Internet-Draft "Machine-Readable Training Preference Signal", and reporting on Meta's commercial terms.