On October 30, Kevin Kelley, a technical lead for standards at Anthropic, published an Internet-Draft for a single robots.txt line that tells AI companies a site does not want its content used for training, and Google, OpenAI and Microsoft said the same day that they would honor it. Three outlets have since covered it, and they describe three different stories: a technical mechanism, a symbolic gesture and a process dispute. Read together, they show that the signal settles one publisher decision and leaves the more expensive one, whether to appear in AI answers, with each company.
What the signal is
The draft, written by K. Kelley of Anthropic, binds a preference to paths by adding an ordinary robots.txt group keyed to a policy token, __usage_ai-training. Its own example disallows the whole site and allows the /press-kit/ folder. The draft says it introduces no new directive names and no new grammar, so existing parsers ignore the group as they would any user agent they do not recognize.
Google’s Search Central blog, posted by Search Advocate John Mueller, says the control “will help websites express their AI training preferences across multiple AI providers in a single step”. If a site uses the token and does not name Google-Extended, Google plans to treat that as a Google-Extended disallow. Microsoft’s Bing blog, written by Principal Product Manager Krishna Madhavan, opens with the stance behind its support: “Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”
Three accounts, three emphases
TheNextWeb’s report by Ana Maria Constantin treats it as a mechanics and timeline story. It sets out who starts when: Anthropic has begun work, OpenAI says it aims to implement the signal “in the coming months”, Google says 2027, and Microsoft honors the new preferences from February 1, 2027. It also lists what the signal does not do. The draft describes a preference and gives no instruction about crawling. A site with no such group has expressed no preference. Private agreements between a site and an AI company can override it. And the draft is an individual submission, which the IETF has not endorsed. TheNextWeb adds that the draft thanks staff from the Financial Times, the BBC, Bertelsmann and the European Publishers Council for shaping the discussion, and that Microsoft said its new preferences “do not establish licensing or payment arrangements”.
Thurrott.com’s post by Paul Thurrott is short and makes a judgment. He calls the change “largely symbolic” because it arrives years after the training it addresses, and he places Google, Microsoft and OpenAI in 2027. That last point differs from TheNextWeb, which quotes OpenAI saying it aims to act “in the coming months”, a phrase that carries no date.
Digiday’s explainer by Sara Guaglione spends its length on enforcement and process. Paul Bannister, chief strategy officer at Raptive, called the token “window dressing” and said it does not address data that reaches AI companies through third-party sellers or open crawls. Alex Springer, technical lead at SPUR, a coalition of publishers working on AI standards, objected to the order of events: the four companies announced support for a draft that the IETF’s AI Preferences working group has not yet reviewed. Digiday also notes that the token covers training and not inference, the retrieval step that has become core to most recent AI licensing deals, and that Microsoft was the only backer to address what blocking would cost in AI visibility.
Where the accounts disagree
The disagreement is about what kind of event this is. TheNextWeb reports a working mechanism with a published schedule. Thurrott reads it as late and symbolic. Digiday’s sources read it as a question about who gets to define a standard and whether anyone checks compliance. The accounts agree on the facts, and the draft, the Google post and the Bing post support each version. They put different weights on the same facts, and a publisher’s reading depends on which weight it already holds.
The accounts also differ on timing. Thurrott’s 2027 for all three later adopters is accurate for Google and Microsoft, whose own posts say so. For OpenAI, the only wording in these accounts is “in the coming months”, and nobody has published a date.
What the three accounts add up to
In our reading, the pieces describe an agreement about one publisher decision, which is what to say about future model training. The token settles that statement across four companies in one line, and that removes work for a site that today maintains separate rules for each crawler. Microsoft’s post says its preferences let a publisher choose training, grounding or both while remaining eligible for Bing search results, and the cost it names attaches to the grounding choice.
The grounding decision sits outside the token, and the primaries show it. Microsoft adds a separate page-level AI Grounding preference, with a Disallow setting and a narrower DisallowExtended setting that keeps Microsoft’s own experiences, such as Copilot, and excludes grounding services it sells to others. Its post states the cost directly: “you will see reduced brand visibility in AI experiences.” If a site expresses no grounding preference, Microsoft says the content may be used to inform generated responses. Google’s post makes a parallel offer, saying sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps.
Digiday’s second report that day, by Jessica Davies and Seb Joseph, shows what that trade-off looks like in practice. Google’s separate Search Console control for excluding content from its AI features followed the UK Competition and Markets Authority’s first conduct requirement on Google, and Google opened it to sites worldwide on August 31. The control is bundled, so publishers cannot leave individual features. Page-level grounding controls are due by March 3, 2027, and click data that Google must give UK publishers is due by December 3. Chris Dicker, chief executive of Candr Media, which owns Trusted Reviews, told Digiday that “everyone is holding off” on the new controls until that data arrives.
So the sequence for a publisher runs in a particular order. The training token arrives first, from February 2027 at Microsoft and during 2027 at Google. On Google’s controls, the visibility decision waits for click data that shows what appearing in AI answers is worth. Licensing, the third decision, is outside the token entirely, and Microsoft says so in terms. Our earlier reporting on Google’s pilot to pay publishers for AI answers covers one version of that third decision, and our analysis of bot-scraping counts covers why sources disagree on crawler traffic. For the crawler side of the same problem, see our report on stealth AI crawlers and the open web.
What it means for the marketing leader
Marketing teams sit on both sides of this. Most run a website with a robots.txt file, and most also depend on being cited in the answers that publishers’ content feeds.
Three practical steps follow. First, treat your own robots.txt as a decision on record. Check which AI crawlers you already name, and decide whether your own pages should carry the training token once the companies honor it, which states a preference about training only. Second, separate the training question from the grounding question in any policy you write, because Microsoft already treats them as different settings and Google’s visibility trade-off turns on the second. Third, put two dates in the calendar: December 3, when Google owes UK publishers click data, and February 1, 2027, when Microsoft starts honoring the new preferences.
There is also a measurement consequence for brand and PR teams. If publishers opt out of grounding, Microsoft warns that brand visibility in AI experiences falls. Brand mentions that arrive through publisher content in AI answers are then partly a function of choices publishers make after December. Track which publishers carry your coverage, and track whether that coverage still shows up in AI answers after those choices land, because the token itself will not tell you.
Source: IETF Internet-Draft, Machine-Readable Training Preference Signal
