AI law and the newsroom, October 30: four AI companies back one robots.txt token, and the EU rules that give such a line legal force
One issue of AI law as it reaches journalism: OpenAI, Google, Microsoft and Anthropic say they will read one shared robots.txt token as a signal about AI training, on four dates running into 2027, while the EU's AI Act code already makes such a line a compliance question for the same four companies.
This issue covers 30 October 2026, the day since the last one, in two items. Four AI companies said they will read one shared line in robots.txt as a signal about AI training. In Brussels, a code of practice those same four have signed already sets out what such a line means.
Two questions run through both. What would writing that line change for a newsroom, and on what date? And what turns a preference in a text file into something a regulator can check?
Four AI companies back one token in robots.txt
OpenAI, Google, Microsoft and Anthropic each published a commitment on Friday 30 October to honour one shared signal in robots.txt, the file a website uses to state what automated visitors may do with its pages. 123
The signal is one word in the User-Agent line of an ordinary robots.txt group:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Read by the matching rules robots.txt already uses, that group states a preference against AI training on every path except
/press-kit/, where it states a preference for it. 4 The token comes from an Internet-Draft published the same day by Kevin Kelley, a technical lead for standards at Anthropic. 4Until now, a publisher wanting out of training had to name each company's crawler or proprietary token and keep a separate rule for every provider. 2 One group now carries the same preference to every provider that has agreed to read it, and the preference can be set path by path. 2
Two properties of the file decide whether the line works. The draft's matching rules exclude wildcard groups from the policy token, so a site whose only rule is
User-agent: * has left the training question unanswered. 4 A file without the token leaves Allow in place: every provider in Friday's announcements treats Allow as the default. 34Who reads it, and from when:
- Anthropic. Kelley told the IETF working group that Anthropic "has already started implementing support for this draft". 5 The Verge reported Anthropic as the one company already applying the signal. 6
- OpenAI. "We aim to implement the signal in the coming months", with its existing crawler controls continuing alongside. 1
- Google. Support planned for 2027. Where the token appears and Google-Extended, its own training control, goes unnamed in the same file, Google plans to read the token as a disallow of Google-Extended. A site that explicitly allows Google-Extended stays eligible for answers and clicks through grounding in Gemini Apps. 2
- Microsoft. Reads the new preferences from 1 February 2027, as its crawler reaches each page. 3 It also adds two settings the shared draft leaves undefined: a page-level AI Training preference that overrides robots.txt, and an AI Grounding preference with a new value,
DisallowExtended, that keeps content out of grounding services Microsoft supplies to other companies. 3
No statute requires the token: these are company commitments, made around a proposed standard.
The first place one of them meets an existing duty is Britain, where a conduct requirement in force since 3 June 2026 obliges Google to let publishers opt out of their content powering AI features in search and out of fine-tuning. Every change is due within nine months, by 3 March 2027. 7 The CMA wrote that duty before this token existed, and has published nothing saying whether one line of robots.txt satisfies it.
Microsoft's announcement also carries the deadline that will catch publishers who do nothing. Its older tags,
NoArchive and NoCache, are read today as requests about training and grounding. After 1 March 2027 Microsoft stops reading them for either purpose, and a site whose wishes are recorded only in those tags falls back to Allow. 3A partial migration carries the sharper risk. Microsoft says that once it detects a new AI Training or AI Grounding preference on a site, it will disregard any
NoArchive and NoCache preferences still on that site. 3 Migrating one section can therefore switch the old tags off across the whole domain.Microsoft's own steps run in three parts: publish the training preference in robots.txt or on pages, publish the grounding preference page by page, and record a temporary grounding preference in Bing Webmaster Tools. 3
That temporary route opens on 15 December 2026 to sites using the old tags, and a recorded preference lasts until 30 September 2027 or until the page is recrawled, whichever comes first. 3 A published setting takes effect only once Microsoft's crawler next visits the page. 3
The token is a statement about use, and nothing more: Microsoft says the preferences grant no licence, establish no payment arrangement, and guarantee neither inclusion, traffic nor payment. 3 Microsoft and Google both say search eligibility survives the choice. 23
The draft keeps fetching rules where they were. A provider may use the preference to inform crawling, and a site that wants a crawler kept out still writes a rule naming that crawler. 4
As a document, the draft carries the standing of an individual submission. Its cover page records that the Internet Engineering Task Force has yet to endorse it and that it holds no formal standing in the standards process. The draft expires on 3 May 2027. 4 The working group's vocabulary draft remains in progress, and Google and Microsoft both say they will keep working inside it. 23
One company is missing from the list: Meta. Andy Stone, Meta's vice-president of communications, called a shared signal "an important step" and said the company looked forward to working with other AI companies on the standards and their implementation. 6 He stopped short of a commitment that Meta will honour the token. 6
Meta has a hand in the document itself: two of its staff, Malikat Rufai and James Whymark, appear in its acknowledgements. 4
What Brussels already reads into that line
In the European Union, a robots.txt line has had a legal job since 2 August 2025, the date the AI Act's model obligations began to apply to providers of general-purpose AI models. 8
Article 53(1)(c) requires such a provider to identify and comply with a reservation of rights made under Article 4(3) of Directive (EU) 2019/790, "including through state-of-the-art technologies". 9
Article 4(3) supplies the condition. The text and data mining exception applies only where the use "has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online". 10 For a news site, the machine-readable form of that reservation is its robots.txt file.
The Commission's General-Purpose AI Code of Practice says what providers do about it. The Code was published on 10 July 2025, and the Commission and the AI Board have confirmed it as an "adequate voluntary tool" through which a provider can show it complies with the AI Act. 11
Its Copyright chapter, in Measure 1.3, applies whenever a signatory crawls for training. The signatory commits to employ web-crawlers that read and follow instructions expressed in accordance with the Robot Exclusion Protocol (robots.txt), as specified in RFC 9309. 12
The same measure commits them to identify and comply with other machine-readable protocols carrying an Article 4(3) reservation, such as metadata from a standards body or already state of the art. 12
Measure 1.3(3) goes further and encourages signatories to join the discussions that produce such standards. 12 Friday's announcement is a cross-vendor candidate to come out of those discussions, and each provider still sets its own date for reading it.
All four companies behind Friday's announcement are on the Code's signatory list. The Commission's list, last updated on 7 October 2026, names OpenAI, Google, Microsoft and Anthropic. 11 Meta appears nowhere on it. 11
One line in Measure 1.3 answers the fear that an AI-training opt-out costs a publisher its search traffic. Signatories running a search engine are encouraged to take measures so that honouring a reservation carries no direct adverse consequence for the indexing of the reserved pages. 12 That is encouragement only, and Google and Microsoft each make the same promise in their own announcements. 23
The Code remains voluntary: signing it is a choice, and the duty it helps a provider meet is Article 53. 11 Measure 1.3(2) adds that the commitment leaves a rightholder free to reserve rights in any manner it chooses. 12
For a publisher selling into the EU, three rules now line up. A machine-readable reservation keeps content out of the mining exception; Article 53 obliges providers to identify and comply with one; the Code names robots.txt as the place to express it. The shared token is a form of that expression which several providers have promised to read, on four dates running through 2027. 4912
The compliance clock
| Obligation or deadline | Who it binds | Date | Source |
|---|---|---|---|
| European Commission targeted consultation on copyright and technology, extended from 3 November | Anyone responding | 10 November 2026 | European Commission |
| Microsoft: temporary AI grounding transition preference opens in Bing Webmaster Tools | Sites using NoArchive or NoCache on that date | 15 December 2026 | Microsoft |
| Microsoft: AI Training and AI Grounding preferences begin taking effect as pages are crawled | Microsoft, for every site that publishes them | 1 February 2027 | Microsoft |
Microsoft: NoArchive and NoCache no longer read for training or grounding | Publishers relying on those tags | 1 March 2027 | Microsoft |
| CMA conduct requirement for Google search: every change implemented | 3 March 2027, nine months from 3 June 2026 | CMA | |
Internet-Draft draft-kelley-aipref-training-preference-signal-00 expires | The document only; it binds nobody | 3 May 2027 | IETF Datatracker |
| Microsoft: temporary grounding preferences expire | Sites that recorded one | 30 September 2027 | Microsoft |
The next hard dates
The nearest dates are the consultation closing on 10 November and Microsoft's transition window opening on 15 December. 313 The first date on which a published training preference is read by Microsoft is 1 February 2027, and the date by which an old tag must be replaced is 1 March 2027.
Anthropic is already at work, while OpenAI's "coming months" and Google's 2027 leave a publisher able to write the token this week and be heard by one of the four.
Meta's position outlasts this announcement: a welcome for the standard, with the question of honouring it left open. 6
OpenAI used the same day to publish excerpts from briefs filed in the New York copyright litigation, among them one from the United States government, arguing that training a model on published work is fair use. 14 The page is the company's own selection of quotations from filings made by other parties. 14
References
- 1
- 2Supporting a shared AI training control for websites
developers.google.com
- 3
- 4
- 5New draft: draft-kelley-aipref-training-preference-signal-00
mailarchive.ietf.org
- 6
- 7
- 8Article 113: Entry into force and application
ai-act-service-desk.ec.europa.eu
- 9Article 53: Obligations for providers of general-purpose AI models
ai-act-service-desk.ec.europa.eu
- 10
- 11The General-Purpose AI Code of Practice
digital-strategy.ec.europa.eu
- 12
- 13Commission seeks feedback on challenges and way forward in the area of effect of technology on copyright
digital-strategy.ec.europa.eu
- 14Fair use benefits everyone
openai.com
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.