TSEP Technical SEOEvidence Protocol
App

SEO & AI Search Updates

What changes. At the source.

Official announcements. SEO and AI search, their sources and related TSEP controls.

tracked sources
50
retained announcements
181
RSS
Updated:

Filtered announcements.

17 results

Clear allEdit filters
Google Search Central Blog Briefing

Supporting a shared AI training control for websites

Agent policy Crawler access
Impact
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
AI training
Status
Status by provider

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

Today, we're announcing our plans to support a shared AI training control in robots.txt. This will help…

Supporting a shared AI training control for websites

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
5801f586c09d2c5da76bb071305f00a7fdf51c9eba92f56efc6c1ca95a607000
Compared text · Download (gzip) · SHA-256
4d0ab73b960c96b2e07dcad87993171d782634b4d7445cfc52c017723bdd3b16

Existence timestamped by DigiCert on · Verify timestamp

Updated common crawlers page to mention future AI training control support

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
2b1c7949a09467740495061478274258fd6e13e1c8506f0ca7ffaf12aa6a78d0

Existence timestamped by DigiCert on · Verify timestamp

Your content, your choice: clearer preferences for training and grounding

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
485728a612d7fe221c10c80c709871afbaef1335aa9ab3662098b3de2dece1d7
Compared text · Download (gzip) · SHA-256
e34ff3d5f65c816f914b4f4bfe6047188288180770f908d1172ca6c818b4a620

Existence timestamped by DigiCert on · Verify timestamp

Making it easier for websites to express AI training preferences

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
a0a2b9bc5914cb528f31cab6f8f73f9524f6b14089f775fc4a80b24d0e95982e
Compared text · Download (gzip) · SHA-256
c630dfb7fc316a8086e2e09a8ecdda7c10826c11a57b7e70af5da12c0a3be83c

Existence timestamped by DigiCert on · Verify timestamp

How Anthropic honors website preferences for AI training

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122

Existence timestamped by DigiCert on · Verify timestamp

Machine-Readable Training Preference Signal

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-06

Source capture · Download (gzip) · SHA-256
50ae1b9b63e6e1abe8f3f01e58981fb5c36a77745d8c381436d4042106dfa625
Compared text · Download (gzip) · SHA-256
81b3855f3dd3a5205aedcc9bb5c1427a1b331ae6b0a28583a286947756faeb4e

Existence timestamped by DigiCert on · Verify timestamp

Permalink

Google Crawling documentation

Updated guidance on robots.txt specifications

Robots.txt rules Agent policy
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Updated the how Google interprets the robots.txt specification to add recommendation for percent-encoding in rule paths.…

Updated guidance on robots.txt specifications

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
c7bd24d3b0eeab2020ef66c1df8e0552c8f2a67b88ab29c336121cad7f8917d5

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Added Retry-After HTTP header details to the reduce crawl rate documentation

Page response Crawler access Pre- and post-deployment checks
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Restructured the emergency crawl rate reduction section and added information and examples…

Added Retry-After HTTP header details to the reduce crawl rate documentation

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
eadd59a2b306a548b93ba10a3aa36db0ca85e2e67d1c0ec183f2158deceeb7e2

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Generalized the Mediapartners-Google crawler documentation

Robots.txt rules Agent policy
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Updated the Mediapartners-Google documentation to clarify that it affects various ad-related Google products (not just AdSense), and renamed…

Generalized the Mediapartners-Google crawler documentation

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
47b392347d32e7e574386f50cd7a71bebb66618fe580c23c804270e98494e983

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Polished and clarified the crawl budget guide

Robots.txt rules Crawler access
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Polished and clarified the Optimize your crawl budget page to improve clarity, terminology consistency, and flow.…

Polished and clarified the crawl budget guide

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
2f030901b11b68cef7a064fa0b4ce2f97f1be1794705b2c08280e36b773601aa

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Updating the NotebookLM user agent

Robots.txt rules Agent policy
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Updated the NotebookLM user agent to be Google-GeminiNotebook. If you hardcoded the old value in your code,…

Updating the NotebookLM user agent

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
49065e37b4f5f80da6f017ef024e609ac1bbe13f8102f606551ec069f623c5ae

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Corrected the user agent string for Google-InspectionTool user agent

Crawler access
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Updated the Google-InspectionTool user agent string.…

Corrected the user agent string for Google-InspectionTool user agent

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
74a6d1efaa751316f6dc4b8cdc8064beb0083eb79febc6efce538e8c9075ad5e

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Adding Web Bot Auth documentation

Crawler access
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Added documentation on how to authenticate requests with Web Bot Auth.…

Adding Web Bot Auth documentation

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
8358bb85863b179fd77c86a47398a66620fcdcdf64ce27d54c7a0232c214a57d

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Added the Google-Agent user agent

Robots.txt rules Agent policy
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Added the Google-Agent user agent and added the IP ranges for user triggered agents to the documentation on…

Added the Google-Agent user agent

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
86fb5f3ced0d079886a97987a023722d83ab0c25223086d6a1dcaf7fe5297780

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Google Crawling documentation

Added an overview page about Google's web crawling

Crawler access
Evidence
Official publication
First detected
2026-11-05
Last seen in the source
2026-11-12
Topic
Crawling and indexing
Status
Documentation updated

Added during the historical import.

Shortened excerpt · normalized text, not verbatim.

What: Added a new page about how Google's crawling works.…

Added an overview page about Google's web crawling

Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.

First capture · · Passage last seen on 2026-11-12

Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653
Compared text · Download (gzip) · SHA-256
22de61d85ab8ecae3fe435653d3e87f1566b3c5514a02ae860c28192da4e2a95

Existence timestamped by DigiCert on · Verify timestamp

Related documentation · Permalink

Reference data

The registers.

Coverage check

What the feed missed.

2 / 3

events found

67 %

2 / 6 provider mentions found

Reports missed by month-end

Shared AI training preference

Google · Microsoft · OpenAI · Anthropic

Events and comparison sources

An event counts once when any of its official reports enters the collection. Each provider is counted separately. Later backfills preserve the month-end score.

Figures cover the sampled events only, compared with independent publications at the end of each month (UTC).

Directory · 50 tracked sources

Sources and coverage.

Google16 sources
Microsoft · Bing4 sources
AI and other platforms16 sources
  • Amazon search and AI crawlers · Hourly Last successful read ·
  • Anthropic crawlers and access · Hourly Last successful read · 1 archived publication Oct 2026
  • Apple Business local presence release notes · Nightly Last successful read · 5 archived publications May 2026 – Oct 2026
  • Apple Business locations and brands · Nightly Last successful read · 1 archived publication Mar 2026
  • Apple Applebot · Hourly Last successful read ·
  • ByteDance Bytespider and access · Nightly Last successful read ·
  • Common Crawl CCBot and access · Nightly Last successful read ·
  • DuckDuckGo DuckAssistBot · Nightly Last successful read · Last attempt failed ·
  • Grok (xAI) announcements and release notes · Hourly Last successful read · 9 archived publications Dec 2024 – May 2026
  • Grok (xAI) web search and sources · Hourly Last successful read ·
  • Meta crawlers and access · Hourly Last successful read ·
  • Mistral AI announcements · Hourly Last successful read · 6 archived publications Nov 2024 – Sep 2026
  • Mistral AI crawlers and access · Hourly Last successful read ·
  • OpenAI publisher announcements · Hourly Last successful read · 2 archived publications Jul 2026 – Oct 2026
  • OpenAI search crawlers · Hourly Last successful read ·
  • Perplexity search crawlers and access · Hourly Last successful read ·
Standards and institutions11 sources
Outside the default selection3 sources
  • Baidu search resource platform · Nightly Last successful read · 6 archived publications Sep 2026 – Nov 2026 Official source
  • Naver Search & Tech · Nightly Last successful read · 18 archived publications Jul 2026 – Nov 2026 Official source
  • Yandex webmaster news · Nightly Last successful read · 5 archived publications Aug 2026 – Oct 2026 Official source

Available archives vary by source. Changes to undated documentation pages are reviewed before they may appear in the feed.

Primary sources feed the editorial selection. Baidu, Naver and Yandex remain available by filter and in search.

The method

Facts. With context.

  1. Collect

    Critical sources are checked hourly; all sources are checked nightly. One record keeps the original title, published date and sources for each event. Undated documentation changes are held for review. The French and English RSS feeds support WebSub to notify compatible subscribers after publication.

  2. Classify

    A yellow square marks a change that may affect a TSEP check, access rule, diagnosis or deadline. Recurring publications and events remain background information. A yellow triangle identifies background news; a dashed circle means classification is pending. Translations are editorial.

  3. Review for TSEP

    Each month, the review connects announcements to relevant TSEP controls. For each item reviewed, the team decides whether the protocol already covers it, a change needs consideration, or monitoring should continue or close.

Impact explains practical consequences after human review in the displayed language. View adds a distinct editorial position: it is public only after personal approval, with the author’s name and reviewed language. AI-assisted proposals remain in the private workspace.

Report an error