TSEP Technical SEOEvidence Protocol
App

Briefing · AI training and robots.txt

One signal. Four timelines.

The __usage_ai-training token lets a site express its model training preference in robots.txt. Google, Bing, OpenAI and Anthropic announced different timelines. This briefing compares their commitments and the rules each provider applies.

Announcements dated · Updated

Cite this briefing

Briefing timeline

Dates to know.

Sources reviewed on

Google Announced

Early 2027 target

Microsoft · Bing Announced
Temporary preference opens

Bing Webmaster Tools opens temporary grounding preferences for sites using NoArchive or NoCache on this date. Source

Application begins

Bing starts applying the new training and grounding preferences as it crawls. Source

Migration deadline

Migration deadline. After this date, NoArchive and NoCache cease to serve as preferences for either use. Source

Temporary preference ends

Temporary grounding settings expire. For an individual page, the setting ends earlier if Bing recrawls it and finds the new preference. Source

OpenAI Announced

Coming months; no fixed date

Anthropic Rolling out

Since Oct 30, 2026 · gradual rollout

Support by provider.

  • GoogleAnnounced
    Early 2027 targetGoogle plans to map a shared opt-out to a Google-Extended disallow when no explicit Google-Extended rule applies. That mapping would also affect grounding in Gemini Apps and Grounding with Google Search on Vertex AI. An explicit rule takes precedence.
    Official statement · Reviewed
    Evidence and revisions

    Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

    First capture · · Passage last seen on 2026-11-12

    Source capture · Download (gzip) · SHA-256
    1ffcc15ef8fabaf4f131a81be2103415c78a1c85ace17538c0c3dfa7431edb46
    Compared text · Download (gzip) · SHA-256
    5594bbabcbcca82026587ff3fe42db5255605c8fc475a54f31bb687aa0f6c349

    Existence timestamped by DigiCert on · Verify timestamp

    Official article:

    Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

    First capture · · Passage last seen on 2026-11-12

    Source capture · Download (gzip) · SHA-256
    dd73e97c8e5e942f3e48bcc14904951542d429cbe279c25f0decad9d8519e832
    Compared text · Download (gzip) · SHA-256
    e08a69aa04ce4b099030b1935b587bee792c987f5027536eae9d9612514043e0

    Existence timestamped by DigiCert on · Verify timestamp

  • Microsoft · BingAnnounced
    Publisher transition opens December 15, 2026Sites already using NoArchive or NoCache on December 15 can set a temporary grounding preference in Bing Webmaster Tools. For training, the page-level preference will take precedence over robots.txt.View all four Bing dates
    Official statement · Reviewed
    Evidence and revisions

    Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

    First capture · · Passage last seen on 2026-11-12

    Source capture · Download (gzip) · SHA-256
    989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b
    Compared text · Download (gzip) · SHA-256
    a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c

    Existence timestamped by DigiCert on · Verify timestamp

  • OpenAIAnnounced
    Coming months; no fixed dateOpenAI announces future implementation. GPTBot and current controls remain in place. The announcement gives no switch-on date.
    Official statement · Reviewed
    Evidence and revisions

    Excerpt of the article read on 2026-11-05 · We aim to implement the signal in the coming months

    SHA-256 · 21d97738ac06b28a6bb90449075ef55d4d0722625aef32d3bcd5986f19aa094d

    The capture below is of the official feed. It does not reverify the timeline in the article.

    Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

    First capture · · Passage last seen on 2026-11-12

    Source capture · Download (gzip) · SHA-256
    a0a2b9bc5914cb528f31cab6f8f73f9524f6b14089f775fc4a80b24d0e95982e
    Compared text · Download (gzip) · SHA-256
    1256781c54c61b681c4f5104722fe94136795d6c5ea943f2290acbb85ef487a2

    Existence timestamped by DigiCert on · Verify timestamp

  • AnthropicRolling out
    Since Oct 30, 2026 · gradual rolloutAnthropic says recognition has begun. A rule naming ClaudeBot takes precedence for the path concerned. The provider describes a gradual rollout across its services.
    Official statement · Reviewed
    Evidence and revisions

    Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

    First capture · · Passage last seen on 2026-11-12

    Source capture · Download (gzip) · SHA-256
    e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122
    Compared text · Download (gzip) · SHA-256
    184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b

    Existence timestamped by DigiCert on · Verify timestamp

Related controls.

Agent policy

Identify which agents are allowed for search, training and user requests.

Open TS42 in TSEP Control
Crawler access

Test access for permitted crawlers and cross-check it against server logs.

Open TS43 in TSEP Control

Correction history

· 2026-11-05.2
The Bing timeline previously listed two dates. Added the December 15, 2026 opening and the September 30, 2027 expiry limit. Clarified how a shared opt-out mapped to Google-Extended could affect Gemini grounding.

blogs.bing.com developers.google.com developers.google.com

Cite this briefing

Edikka · TSEP. « AI training and robots.txt: the Google, Bing, OpenAI and Anthropic timelines ». Edition 2026-11-05.2. TSEP-VEILLE-2026-10-30-01.

AI-assisted editorial writing. Review signed by Bertrand Morel · 2026-11-06.

IETF AIPREF · travaux du groupe · A comparison identified a presentation-only change, with no change to the reviewed documentary passage. The review retains its original date. View changes and evidence

Compare AI directives and crawlers