# AI crawlers: list, uses and robots.txt rules

Edition: 2026-11-06.2 · Edikka — TSEP · tsep-ai-crawlers

[HTML](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt) · [JSON](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt.json) · [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

TSEP factual compilation and editorial classifications. Source documents retain their original terms.

## Method and review

AI-assisted documentary readings based on archived official sources. “Documented” describes what a provider publishes, not observed behavior. A reading is withdrawn if a supporting source passage changes.

Review dates, capture dates and effective dates are distinct. An omitted effective date or delay means the supporting source does not establish it; it does not mean immediate application or no delay.

Examples are targeted: integrate them with existing rules and check their scope before publishing. A User-Agent can be spoofed; its name alone cannot attribute a request.

SHA-256 hashes apply to decompressed captures. Compared text is a normalized extraction. Independent timestamps establish the existence of a manifest, not the truth of a document or crawler compliance.

Human review of this edition: pending.

SHA-256 (editorial content): `3b33308b3a4e63190d1cab529c07f318dc98d6035445dfd99863933f26fa40f9`

## Shared AI training preference

[usage-ai-training](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#usage-ai-training)

Expresses a generative-training opt-out to providers implementing the signal.

User-agent: * does not express this preference. Without a dedicated group, no shared preference is stated. Provider-specific rules and private agreements may take precedence. The token does not block HTTP requests.

Individual IETF submission, version -00 dated 30 October 2026; expires 3 May 2027. Not an adopted standard.

### Opt out of training site-wide

robots.txt

```text
User-agent: __usage_ai-training
Disallow: /
```


### Google — planned

Document review: 2026-11-06

Early 2027 target

Google plans to map a shared opt-out to a Google-Extended disallow when no explicit Google-Extended rule applies. That mapping would also affect grounding in Gemini Apps and Grounding with Google Search on Vertex AI. An explicit rule takes precedence.

An explicit Google-Extended rule retains priority.

[Official source](https://developers.google.com/search/blog/2026/10/ai-training-control)

Captured: 2026-11-05T17:05:01+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/1ffcc15ef8fabaf4f131a81be2103415c78a1c85ace17538c0c3dfa7431edb46.gz) · SHA-256: `1ffcc15ef8fabaf4f131a81be2103415c78a1c85ace17538c0c3dfa7431edb46`
- [passage](https://tsep-protocol.org/en/updates/captures/5594bbabcbcca82026587ff3fe42db5255605c8fc475a54f31bb687aa0f6c349.gz) · SHA-256: `5594bbabcbcca82026587ff3fe42db5255605c8fc475a54f31bb687aa0f6c349`

Captured: 2026-11-05T17:04:55+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/dd73e97c8e5e942f3e48bcc14904951542d429cbe279c25f0decad9d8519e832.gz) · SHA-256: `dd73e97c8e5e942f3e48bcc14904951542d429cbe279c25f0decad9d8519e832`
- [passage](https://tsep-protocol.org/en/updates/captures/e08a69aa04ce4b099030b1935b587bee792c987f5027536eae9d9612514043e0.gz) · SHA-256: `e08a69aa04ce4b099030b1935b587bee792c987f5027536eae9d9612514043e0`

### Microsoft · Bing — planned

Document review: 2026-11-06

Bing starts applying the new training and grounding preferences as it crawls.

Sites already using NoArchive or NoCache on December 15 can set a temporary grounding preference in Bing Webmaster Tools. For training, the page-level preference will take precedence over robots.txt.

The page-level training preference overrides robots.txt.

[Official source](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)

Captured: 2026-11-05T17:05:02+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b.gz) · SHA-256: `989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b`
- [passage](https://tsep-protocol.org/en/updates/captures/a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c.gz) · SHA-256: `a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c`

### OpenAI — planned

Document review: 2026-11-06

Coming months; no fixed date

OpenAI announces future implementation. GPTBot and current controls remain in place. The announcement gives no switch-on date.

Current controls remain; new-signal precedence is unspecified.

[Official source](https://openai.com/index/helping-websites-express-ai-training-preferences)

Captured: 2026-11-05T17:05:00+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/a0a2b9bc5914cb528f31cab6f8f73f9524f6b14089f775fc4a80b24d0e95982e.gz) · SHA-256: `a0a2b9bc5914cb528f31cab6f8f73f9524f6b14089f775fc4a80b24d0e95982e`
- [passage](https://tsep-protocol.org/en/updates/captures/1256781c54c61b681c4f5104722fe94136795d6c5ea943f2290acbb85ef487a2.gz) · SHA-256: `1256781c54c61b681c4f5104722fe94136795d6c5ea943f2290acbb85ef487a2`

### Anthropic — rolling

Document review: 2026-11-06

Since Oct 30, 2026 · gradual rollout

Anthropic says recognition has begun. A rule naming ClaudeBot takes precedence for the path concerned. The provider describes a gradual rollout across its services.

A rule naming ClaudeBot overrides the shared signal.

[Official source](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Captured: 2026-11-05T17:04:54+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122.gz) · SHA-256: `e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122`
- [passage](https://tsep-protocol.org/en/updates/captures/184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b.gz) · SHA-256: `184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## GPTBot

[gptbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#gptbot)

Opts crawled content out of OpenAI generative foundation-model training.

ChatGPT search is configured separately using OAI-SearchBot.

### Disallow this crawler

robots.txt

```text
User-agent: GPTBot
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
```

- [IP](https://openai.com/gptbot.json)

### OpenAI — documented

Document review: 2026-11-06

[Official source](https://developers.openai.com/api/docs/bots)

Captured: 2026-11-05T17:04:53+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/78caa706896816acebafc0b949ca1680411273c4a24bea157b1dd8bf4333252a.gz) · SHA-256: `78caa706896816acebafc0b949ca1680411273c4a24bea157b1dd8bf4333252a`
- [passage](https://tsep-protocol.org/en/updates/captures/b7c5a2b25a2fd1fcd65f7a22fe4cc23b7be11057953d41a36a3e9093c330b112.gz) · SHA-256: `b7c5a2b25a2fd1fcd65f7a22fe4cc23b7be11057953d41a36a3e9093c330b112`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## ClaudeBot

[claudebot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#claudebot)

Signals that future content should be excluded from Claude training datasets.

Does not apply the same choice to Claude-SearchBot or Claude-User. Dedicated ClaudeBot rules override the shared signal.

### Disallow this crawler

robots.txt

```text
User-agent: ClaudeBot
Disallow: /
```

- [IP](https://claude.com/crawling/bots.json)

### Anthropic — documented

Document review: 2026-11-06

[Official source](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Captured: 2026-11-05T17:04:54+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122.gz) · SHA-256: `e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122`
- [passage](https://tsep-protocol.org/en/updates/captures/184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b.gz) · SHA-256: `184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## ai-training

[bing-ai-training](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#bing-ai-training)

Opts the page out of generative training.

The page preference overrides the robots.txt signal. Bingbot must be able to read it; Allow does not lift a crawl block.

Application announced from 1 February 2027, as pages are crawled.

### Page HTML tag

HTML

```text
<meta name="Bingbot" content="ai-training:disallow">
```

### Alternative: HTTP header

HTTP

```text
X-Robots-Tag: Bingbot: ai-training:disallow
```


### Microsoft · Bing — planned

Document review: 2026-11-06

[Official source](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)

Captured: 2026-11-05T17:05:02+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b.gz) · SHA-256: `989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b`
- [passage](https://tsep-protocol.org/en/updates/captures/a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c.gz) · SHA-256: `a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## Google-Extended

[google-extended](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#google-extended)

Opts out of Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

Does not affect Search inclusion or ranking, AI Overviews or AI Mode. This token does not identify a separate HTTP crawler.

### Disallow Google-Extended uses

robots.txt

```text
User-agent: Google-Extended
Disallow: /
```


### Google — documented

Document review: 2026-11-06

[Official source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

Captured: 2026-11-05T17:05:01+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/1ffcc15ef8fabaf4f131a81be2103415c78a1c85ace17538c0c3dfa7431edb46.gz) · SHA-256: `1ffcc15ef8fabaf4f131a81be2103415c78a1c85ace17538c0c3dfa7431edb46`
- [passage](https://tsep-protocol.org/en/updates/captures/5594bbabcbcca82026587ff3fe42db5255605c8fc475a54f31bb687aa0f6c349.gz) · SHA-256: `5594bbabcbcca82026587ff3fe42db5255605c8fc475a54f31bb687aa0f6c349`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## ai-grounding

[bing-ai-grounding](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#bing-ai-grounding)

Disallow opts out of grounding. DisallowExtended limits it to Microsoft-operated experiences, excluding Microsoft grounding services for third parties.

Page-level only, never in robots.txt. Independent of training, with no change to eligibility for traditional Bing results.

Application announced from 1 February 2027, as pages are crawled.

### Disallow all grounding

HTML

```text
<meta name="Bingbot" content="ai-grounding:disallow">
```

### Limit to Microsoft-operated experiences

HTML

```text
<meta name="Bingbot" content="ai-grounding:disallowextended">
```

### Alternative: HTTP header

HTTP

```text
X-Robots-Tag: Bingbot: ai-grounding:disallowextended
```


### Microsoft · Bing — planned

Document review: 2026-11-06

[Official source](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)

Captured: 2026-11-05T17:05:02+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b.gz) · SHA-256: `989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b`
- [passage](https://tsep-protocol.org/en/updates/captures/a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c.gz) · SHA-256: `a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## NoArchive / NoCache

[noarchive-nocache](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#noarchive-nocache)

Bing’s legacy AI preferences should be replaced before 1 March 2027.

For eligible sites: temporary grounding choice from 15 December 2026; new rules read from 1 February; temporary choice ends at recrawl or on 30 September 2027.

Migration: use the ai-training and ai-grounding records.


### Microsoft · Bing — transition

Document review: 2026-11-06

[Official source](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)

Captured: 2026-11-05T17:05:02+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b.gz) · SHA-256: `989edfcce11f91b5c34f54102d7a431c1cd62747ff6b865281600ee1cc9a5f1b`
- [passage](https://tsep-protocol.org/en/updates/captures/a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c.gz) · SHA-256: `a577becb8222f398914b1cb231901636213618d580bd70cd7f2819777165d28c`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## Search Console

[search-console](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#search-console)

The exclusion setting removes links and content from AI Overviews, AI Mode and generative features in Discover.

Does not control training or ranking/inclusion in other Search results. Child properties may inherit the parent setting.

Worldwide rollout reported complete on 31 August 2026.

Search Console → Settings → Search generative AI → Exclude my site’s links and content.

Stated delay: Generally a few days; 1–2 days after activation, sometimes longer due to caches.


### Google — documented

Document review: 2026-11-06

[Official source](https://support.google.com/webmasters/answer/16908024?hl=en)

Captured: 2026-11-05T17:05:04+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/9265c70566faffc412699db6d856852bf28a2269c01ddcbd17cff4c1834b9cf3.gz) · SHA-256: `9265c70566faffc412699db6d856852bf28a2269c01ddcbd17cff4c1834b9cf3`
- [passage](https://tsep-protocol.org/en/updates/captures/eaa354880f18bf54448d9ab868a8080f33d89987939a4222bf4aa2d452afe412.gz) · SHA-256: `eaa354880f18bf54448d9ab868a8080f33d89987939a4222bf4aa2d452afe412`



## OAI-SearchBot

[oai-searchbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#oai-searchbot)

Opt-out removes the site from ChatGPT search answers, although navigational links may remain.

Does not configure GPTBot. Allowing search also requires working network access.

Stated delay: About 24 hours for search systems.

### Disallow this crawler

robots.txt

```text
User-agent: OAI-SearchBot
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
```

- [IP](https://openai.com/searchbot.json)

### OpenAI — documented

Document review: 2026-11-06

[Official source](https://developers.openai.com/api/docs/bots)

Captured: 2026-11-05T17:04:53+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/78caa706896816acebafc0b949ca1680411273c4a24bea157b1dd8bf4333252a.gz) · SHA-256: `78caa706896816acebafc0b949ca1680411273c4a24bea157b1dd8bf4333252a`
- [passage](https://tsep-protocol.org/en/updates/captures/b7c5a2b25a2fd1fcd65f7a22fe4cc23b7be11057953d41a36a3e9093c330b112.gz) · SHA-256: `b7c5a2b25a2fd1fcd65f7a22fe4cc23b7be11057953d41a36a3e9093c330b112`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Claude-SearchBot

[claude-searchbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#claude-searchbot)

Opt-out prevents indexing intended to improve Claude search results.

May reduce search visibility and accuracy; training and user-requested access are configured separately.

### Disallow this crawler

robots.txt

```text
User-agent: Claude-SearchBot
Disallow: /
```

- [IP](https://claude.com/crawling/bots.json)

### Anthropic — documented

Document review: 2026-11-06

[Official source](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Captured: 2026-11-05T17:04:54+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122.gz) · SHA-256: `e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122`
- [passage](https://tsep-protocol.org/en/updates/captures/184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b.gz) · SHA-256: `184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Applebot-Extended

[applebot-extended](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#applebot-extended)

Opts Applebot data out of foundation-model training.

Does not block Applebot or Apple search. This preference token is not a standalone crawler.

### Opt out of Apple training

robots.txt

```text
User-agent: Applebot-Extended
Disallow: /
```


### Apple — documented

Document review: 2026-11-06

[Official source](https://support.apple.com/en-us/119829)

Captured: 2026-11-05T17:05:18+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/30c9ef308e140287d2a6310879414df7f87a65314234b51d00bd971d6f4daad3.gz) · SHA-256: `30c9ef308e140287d2a6310879414df7f87a65314234b51d00bd971d6f4daad3`
- [passage](https://tsep-protocol.org/en/updates/captures/58cfc33eeebcef38f90dd04942c45862a15e4038451adfad5e4f7950709463cd.gz) · SHA-256: `58cfc33eeebcef38f90dd04942c45862a15e4038451adfad5e4f7950709463cd`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## MistralAI-Training

[mistralai-training](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#mistralai-training)

Opts out of MistralAI-Training collection for generative training.

MistralAI-Index and MistralAI-User retain their own settings.

### Disallow this crawler

robots.txt

```text
User-agent: MistralAI-Training
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)
```


### Mistral — documented

Document review: 2026-11-06

[Official source](https://docs.mistral.ai/robots)

Captured: 2026-11-06T05:24:07+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/b4539f23004f3dae36adfc1142c7f09584bf365d5f3a94cb3d8ab1d90f2a3602.gz) · SHA-256: `b4539f23004f3dae36adfc1142c7f09584bf365d5f3a94cb3d8ab1d90f2a3602`
- [passage](https://tsep-protocol.org/en/updates/captures/7ae2e284a1dfaf3bcdbf76c301a9ed26f96ec1f5c040ffe90f7a3dc84fffadd4.gz) · SHA-256: `7ae2e284a1dfaf3bcdbf76c301a9ed26f96ec1f5c040ffe90f7a3dc84fffadd4`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## meta-externalagent

[meta-externalagent](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#meta-externalagent)

Disallows this agent’s collection for training and Meta product improvement.

Its scope also includes product indexing; it is not an isolated training preference.

### Disallow this crawler

robots.txt

```text
User-agent: meta-externalagent
Disallow: /
```

User-Agent (documented example):

```text
meta-externalagent/1.1
```


### Meta — documented

Document review: 2026-11-06

[Official source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

Captured: 2026-11-06T09:56:25+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/a4d5f103e841ac5b6fd7b2abf86a42a6a93c17dfdc072abed0466e5ba3e38008.gz) · SHA-256: `a4d5f103e841ac5b6fd7b2abf86a42a6a93c17dfdc072abed0466e5ba3e38008`
- [passage](https://tsep-protocol.org/en/updates/captures/5210cb46fad2ff54b5038bdb0e648c593311a4ffc120a40f8073874cc2484fe9.gz) · SHA-256: `5210cb46fad2ff54b5038bdb0e648c593311a4ffc120a40f8073874cc2484fe9`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Amazonbot

[amazonbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#amazonbot)

Disallows Amazonbot crawling, used for its products and potentially training.

Amzn-SearchBot and Amzn-User settings are independent.

Stated delay: About 24 hours stated by Amazon; no guaranteed per-page delay.

### Disallow this crawler

robots.txt

```text
User-agent: Amazonbot
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
```

- [IP](https://developer.amazon.com/amazonbot/ip-addresses/)

### Amazon — documented

Document review: 2026-11-06

[Official source](https://developer.amazon.com/docs/amazonbot/amazonbot.html)

Captured: 2026-11-06T09:56:24+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6.gz) · SHA-256: `ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6`
- [passage](https://tsep-protocol.org/en/updates/captures/215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4.gz) · SHA-256: `215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## PerplexityBot

[perplexitybot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#perplexitybot)

Disallows crawling for Perplexity search.

Perplexity-User is separate; the provider does not describe this bot as a training crawler.

Stated delay: Up to 24 hours for changes to take effect, according to Perplexity. Stated delay, not measured by TSEP.

### Disallow this crawler

robots.txt

```text
User-agent: PerplexityBot
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
```

- [IP](https://www.perplexity.ai/perplexitybot.json)

### Perplexity — documented

Document review: 2026-11-06

[Official source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

Captured: 2026-11-06T09:56:23+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/8b0b02c3c0b8529c60eb3a0e9e29a2e41cf7854406e9e1fdbe59fc2a1d3134b1.gz) · SHA-256: `8b0b02c3c0b8529c60eb3a0e9e29a2e41cf7854406e9e1fdbe59fc2a1d3134b1`
- [passage](https://tsep-protocol.org/en/updates/captures/bcb760d7c84755ab6da10bcaa5a72d7f81f7d2ff02acb1d2fa118a978b0f88ef.gz) · SHA-256: `bcb760d7c84755ab6da10bcaa5a72d7f81f7d2ff02acb1d2fa118a978b0f88ef`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## MistralAI-Index

[mistralai-index](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#mistralai-index)

Disallows automated Mistral search indexing.

This crawler is not used for generative-model training.

### Disallow this crawler

robots.txt

```text
User-agent: MistralAI-Index
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
```

- [IP](https://mistral.ai/mistralai-index-ips.json)

### Mistral — documented

Document review: 2026-11-06

[Official source](https://docs.mistral.ai/robots)

Captured: 2026-11-06T05:24:07+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/b4539f23004f3dae36adfc1142c7f09584bf365d5f3a94cb3d8ab1d90f2a3602.gz) · SHA-256: `b4539f23004f3dae36adfc1142c7f09584bf365d5f3a94cb3d8ab1d90f2a3602`
- [passage](https://tsep-protocol.org/en/updates/captures/7ae2e284a1dfaf3bcdbf76c301a9ed26f96ec1f5c040ffe90f7a3dc84fffadd4.gz) · SHA-256: `7ae2e284a1dfaf3bcdbf76c301a9ed26f96ec1f5c040ffe90f7a3dc84fffadd4`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## meta-webindexer

[meta-webindexer](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#meta-webindexer)

Disallows indexing for Meta AI search and citations.

Does not replace settings for other Meta agents.

### Disallow this crawler

robots.txt

```text
User-agent: meta-webindexer
Disallow: /
```

User-Agent (documented example):

```text
meta-webindexer/1.1
```


### Meta — documented

Document review: 2026-11-06

[Official source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

Captured: 2026-11-06T09:56:25+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/a4d5f103e841ac5b6fd7b2abf86a42a6a93c17dfdc072abed0466e5ba3e38008.gz) · SHA-256: `a4d5f103e841ac5b6fd7b2abf86a42a6a93c17dfdc072abed0466e5ba3e38008`
- [passage](https://tsep-protocol.org/en/updates/captures/5210cb46fad2ff54b5038bdb0e648c593311a4ffc120a40f8073874cc2484fe9.gz) · SHA-256: `5210cb46fad2ff54b5038bdb0e648c593311a4ffc120a40f8073874cc2484fe9`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Amzn-SearchBot

[amzn-searchbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#amzn-searchbot)

Disallows crawling dedicated to search in Amazon products.

Without a dedicated rule, Amazon may use rules for other allowed search crawlers. This bot does not crawl for generative training.

Stated delay: About 24 hours stated by Amazon; no guaranteed per-page delay.

### Disallow this crawler

robots.txt

```text
User-agent: Amzn-SearchBot
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
```

- [IP](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/)

### Amazon — documented

Document review: 2026-11-06

[Official source](https://developer.amazon.com/docs/amazonbot/amazonbot.html)

Captured: 2026-11-06T09:56:24+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6.gz) · SHA-256: `ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6`
- [passage](https://tsep-protocol.org/en/updates/captures/215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4.gz) · SHA-256: `215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Applebot

[applebot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#applebot)

Blocking Applebot can remove presence in Apple search.

For training only, use Applebot-Extended; for AI broad world knowledge answers, use nosnippet. Without Applebot rules, Apple uses Googlebot rules if present.

### Disallow this crawler

robots.txt

```text
User-agent: Applebot
Disallow: /
```

- [IP](https://search.developer.apple.com/applebot.json)

### Apple — documented

Document review: 2026-11-06

[Official source](https://support.apple.com/en-us/119829)

Captured: 2026-11-05T17:05:18+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/30c9ef308e140287d2a6310879414df7f87a65314234b51d00bd971d6f4daad3.gz) · SHA-256: `30c9ef308e140287d2a6310879414df7f87a65314234b51d00bd971d6f4daad3`
- [passage](https://tsep-protocol.org/en/updates/captures/58cfc33eeebcef38f90dd04942c45862a15e4038451adfad5e4f7950709463cd.gz) · SHA-256: `58cfc33eeebcef38f90dd04942c45862a15e4038451adfad5e4f7950709463cd`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Perplexity-User

[perplexity-user](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#perplexity-user)

A robots.txt group does not guarantee exclusion of these user-requested accesses.

User-requested access. Perplexity says these requests generally ignore robots.txt; they are not training crawls.

No snippet is presented as a guaranteed block. Check actual server access.

User-Agent (documented example):

```text
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
```

- [IP](https://www.perplexity.ai/perplexity-user.json)

### Perplexity — documented

Document review: 2026-11-06

[Official source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

Captured: 2026-11-06T09:56:23+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/8b0b02c3c0b8529c60eb3a0e9e29a2e41cf7854406e9e1fdbe59fc2a1d3134b1.gz) · SHA-256: `8b0b02c3c0b8529c60eb3a0e9e29a2e41cf7854406e9e1fdbe59fc2a1d3134b1`
- [passage](https://tsep-protocol.org/en/updates/captures/bcb760d7c84755ab6da10bcaa5a72d7f81f7d2ff02acb1d2fa118a978b0f88ef.gz) · SHA-256: `bcb760d7c84755ab6da10bcaa5a72d7f81f7d2ff02acb1d2fa118a978b0f88ef`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Amzn-User

[amzn-user](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#amzn-user)

A robots.txt group does not guarantee exclusion of these user-requested accesses.

On-demand access to answer a user, without training. Amazon says these requests may not follow every robots.txt rule.

No snippet is presented as a guaranteed block. Check actual server access.

Stated delay: About 24 hours stated by Amazon; no guaranteed per-page delay.

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36
```

- [IP](https://developer.amazon.com/amazonbot/live-ip-addresses/)

### Amazon — documented

Document review: 2026-11-06

[Official source](https://developer.amazon.com/docs/amazonbot/amazonbot.html)

Captured: 2026-11-06T09:56:24+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6.gz) · SHA-256: `ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6`
- [passage](https://tsep-protocol.org/en/updates/captures/215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4.gz) · SHA-256: `215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## MistralAI-User

[mistralai-user](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#mistralai-user)

Controls sites accessible to Vibe user requests.

Neither automated web crawling nor training collection.

### Disallow this crawler

robots.txt

```text
User-agent: MistralAI-User
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
```

- [IP](https://mistral.ai/mistralai-user-ips.json)

### Mistral — documented

Document review: 2026-11-06

[Official source](https://docs.mistral.ai/robots)

Captured: 2026-11-06T05:24:07+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/b4539f23004f3dae36adfc1142c7f09584bf365d5f3a94cb3d8ab1d90f2a3602.gz) · SHA-256: `b4539f23004f3dae36adfc1142c7f09584bf365d5f3a94cb3d8ab1d90f2a3602`
- [passage](https://tsep-protocol.org/en/updates/captures/7ae2e284a1dfaf3bcdbf76c301a9ed26f96ec1f5c040ffe90f7a3dc84fffadd4.gz) · SHA-256: `7ae2e284a1dfaf3bcdbf76c301a9ed26f96ec1f5c040ffe90f7a3dc84fffadd4`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## meta-externalfetcher

[meta-externalfetcher](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#meta-externalfetcher)

A robots.txt group does not guarantee exclusion of these user-requested accesses.

Retrieves links on demand and evaluates agent capabilities. Meta says some requests may bypass robots.txt.

No snippet is presented as a guaranteed block. Check actual server access.

User-Agent (documented example):

```text
meta-externalfetcher/1.1
```


### Meta — documented

Document review: 2026-11-06

[Official source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

Captured: 2026-11-06T09:56:25+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/a4d5f103e841ac5b6fd7b2abf86a42a6a93c17dfdc072abed0466e5ba3e38008.gz) · SHA-256: `a4d5f103e841ac5b6fd7b2abf86a42a6a93c17dfdc072abed0466e5ba3e38008`
- [passage](https://tsep-protocol.org/en/updates/captures/5210cb46fad2ff54b5038bdb0e648c593311a4ffc120a40f8073874cc2484fe9.gz) · SHA-256: `5210cb46fad2ff54b5038bdb0e648c593311a4ffc120a40f8073874cc2484fe9`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## CCBot

[ccbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#ccbot)

Disallows future compliant Common Crawl crawls.

Does not withdraw already published copies or third-party reuse.

### Disallow this crawler

robots.txt

```text
User-agent: CCBot
Disallow: /
```


### Common Crawl — documented

Document review: 2026-11-06

[Official source](https://commoncrawl.org/ccbot)

Captured: 2026-11-06T09:56:24+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/462fee13c76952805dc093782518a4ee50034c8d73e081cae5177ce55a67d8f9.gz) · SHA-256: `462fee13c76952805dc093782518a4ee50034c8d73e081cae5177ce55a67d8f9`
- [passage](https://tsep-protocol.org/en/updates/captures/51b4b9b75218fa051e37afbbc4fce8506c0f68fb90fa3689e6b3854bb8143cb3.gz) · SHA-256: `51b4b9b75218fa051e37afbbc4fce8506c0f68fb90fa3689e6b3854bb8143cb3`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Content-Signal

[content-signal](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#content-signal)

Declares search, ai-input and ai-train preferences. The documentation also presents an experimental use field.

use=immediate: interact without retention; reference: index, excerpt and link; full: summarize and reproduce. The page is dated 2 November; this does not establish the test start date. Neither network blocking nor universal compliance is guaranteed. Examples containing Allow: / must be reconciled with existing exclusions.

### Cloudflare-managed example

robots.txt

```text
User-agent: *
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /
```


### Cloudflare — documented

Document review: 2026-11-06

[Official source](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)

Captured: 2026-11-06T09:56:27+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/ff7d1b9bbe5e25c70f7800aeaa2bfb451bc398ecf06d01c6244496d8629ed33f.gz) · SHA-256: `ff7d1b9bbe5e25c70f7800aeaa2bfb451bc398ecf06d01c6244496d8629ed33f`
- [passage](https://tsep-protocol.org/en/updates/captures/bda25ec127fb70eb91f3ddedc9595e123bafde65c2786b6bb4f5732c6c7200e4.gz) · SHA-256: `bda25ec127fb70eb91f3ddedc9595e123bafde65c2786b6bb4f5732c6c7200e4`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## License

[license](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#license)

Associates content with an RSL XML license document.

The license file must exist and define rights. License does not replace Allow/Disallow and depends on clients implementing RSL.

### License discovery example to adapt

robots.txt

```text
User-agent: *
License: https://example.com/rsl-license.xml
```


### RSL — documented

Document review: 2026-11-06

[Official source](https://rslstandard.org/rsl)

Captured: 2026-11-06T09:56:27+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/5667103bb443c15c723911b192bc721ababb27bf1fc76c599e87b995301310de.gz) · SHA-256: `5667103bb443c15c723911b192bc721ababb27bf1fc76c599e87b995301310de`
- [passage](https://tsep-protocol.org/en/updates/captures/9b13e4e1caea2b43a25df581cb878abdbc2bb347c5932d1f6abcb2705c2e114a.gz) · SHA-256: `9b13e4e1caea2b43a25df581cb878abdbc2bb347c5932d1f6abcb2705c2e114a`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## Bytespider

[bytespider](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#bytespider)

Expresses a Bytespider crawl opt-out under ByteDance documentation.

Documentation alone does not establish observed compliance or identify all group traffic.

### Disallow this crawler

robots.txt

```text
User-agent: Bytespider
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
```

User-Agent (documented example):

```text
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
```

User-Agent (documented example):

```text
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
```


### ByteDance — documented

Document review: 2026-11-06

[Official source](https://zhanzhang.toutiao.com/docs/intro/26899)

Captured: 2026-11-06T11:31:52+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/2905dcc31528f09fe690cdc46f45a9867620bfc63231afdc6f9d6e7b54962676.gz) · SHA-256: `2905dcc31528f09fe690cdc46f45a9867620bfc63231afdc6f9d6e7b54962676`
- [passage](https://tsep-protocol.org/en/updates/captures/a67f04988875799a7c2aab5d6c32546adba7820faa9d58f863e558ca457d9000.gz) · SHA-256: `a67f04988875799a7c2aab5d6c32546adba7820faa9d58f863e558ca457d9000`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## ChatGPT-User

[chatgpt-user](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#chatgpt-user)

A robots.txt group does not guarantee exclusion of these user-requested accesses.

Retrieves pages at a ChatGPT or custom GPT user’s request. These requests may not apply robots.txt rules.

No snippet is presented as a guaranteed block. Check actual server access.

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
```

- [IP](https://openai.com/chatgpt-user.json)

### OpenAI — documented

Document review: 2026-11-06

[Official source](https://developers.openai.com/api/docs/bots)

Captured: 2026-11-06T14:55:39+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/b63a02a9eb2ec848efc5dbeb16e63c04cf6d425fb79423bc534267a456c14cca.gz) · SHA-256: `b63a02a9eb2ec848efc5dbeb16e63c04cf6d425fb79423bc534267a456c14cca`
- [passage](https://tsep-protocol.org/en/updates/captures/6d11f6beab02ffe889ac8bfac323d0c69d4e7e5040b7bcc4406a2c49b0239818.gz) · SHA-256: `6d11f6beab02ffe889ac8bfac323d0c69d4e7e5040b7bcc4406a2c49b0239818`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Claude-User

[claude-user](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#claude-user)

Opt-out prevents Claude from retrieving content in response to a user query.

May reduce visibility in these answers; other Claude crawlers remain independent.

### Disallow this crawler

robots.txt

```text
User-agent: Claude-User
Disallow: /
```

- [IP](https://claude.com/crawling/bots.json)

### Anthropic — documented

Document review: 2026-11-06

[Official source](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Captured: 2026-11-05T17:04:54+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122.gz) · SHA-256: `e602ce6895e8e0d41177f00512c3ac06cdde17eac9f78fc43bb905de142cc122`
- [passage](https://tsep-protocol.org/en/updates/captures/184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b.gz) · SHA-256: `184fc8b017b0c1cca4eb7595a8c2b378fb539e3444d02d436ea6f39d41c86c6b`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Googlebot

[googlebot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#googlebot)

Disallows Google’s main search crawler.

No dedicated robots.txt rule targets only AI Overviews or AI Mode: blocking Googlebot affects Google Search as a whole. Use nosnippet, data-nosnippet, max-snippet or noindex to limit display according to the intended scope; the Search Console setting can exclude generative features. A URL blocked from crawling may remain known: robots.txt does not replace noindex.

### Disallow this crawler

robots.txt

```text
User-agent: Googlebot
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
```

- [IP](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)

### Google — documented

Document review: 2026-11-06

[Official source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

Captured: 2026-11-06T14:55:43+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/03b62207ae01bfa409db09177986f4df4b9574d74e6af9fbc0ad5a75bc00481e.gz) · SHA-256: `03b62207ae01bfa409db09177986f4df4b9574d74e6af9fbc0ad5a75bc00481e`
- [passage](https://tsep-protocol.org/en/updates/captures/d8c7efdef95daf256d2a6889b886443324243ef4d2068e587e0deebe69ef5030.gz) · SHA-256: `d8c7efdef95daf256d2a6889b886443324243ef4d2068e587e0deebe69ef5030`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Bingbot

[bingbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#bingbot)

Disallows Bingbot crawling; page-level AI settings cannot be read if access is blocked.

For an AI-use-only opt-out, see ai-training and ai-grounding. A User-Agent header alone does not establish crawler identity.

### Disallow this crawler

robots.txt

```text
User-agent: bingbot
Disallow: /
```

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
```


### Microsoft · Bing — documented

Document review: 2026-11-06

[Official source](https://blogs.bing.com/webmaster/2022/4/Announcing-user-agent-change-for-Bing-crawler-bingbot/)

Captured: 2026-11-06T14:56:26+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/2f5806b31a9569994fff9d3563745fbb25db2fc447e4ead985ed274efd52834f.gz) · SHA-256: `2f5806b31a9569994fff9d3563745fbb25db2fc447e4ead985ed274efd52834f`
- [passage](https://tsep-protocol.org/en/updates/captures/be8b106e6f51822d6d44c2f552906821ed119e7fc4a23bcc2cc94abdbe9df415.gz) · SHA-256: `be8b106e6f51822d6d44c2f552906821ed119e7fc4a23bcc2cc94abdbe9df415`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## DuckAssistBot

[duckassistbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#duckassistbot)

Opt-out stops crawling for DuckDuckGo AI answers after the documented delay.

No stated impact on organic ranking or inclusion. The data is not used to train models.

Stated delay: 72 hours after updating robots.txt.

### Disallow this crawler

robots.txt

```text
User-agent: DuckAssistBot
Disallow: /
```

User-Agent (documented example):

```text
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
```

- [IP](https://duckduckgo.com/duckassistbot.json)

### DuckDuckGo — documented

Document review: 2026-11-06

[Official source](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)

Captured: 2026-11-06T15:01:24+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/7f3a9244de52b91080adc8476e2c30567a78473099136aa3a046b2a49e197f87.gz) · SHA-256: `7f3a9244de52b91080adc8476e2c30567a78473099136aa3a046b2a49e197f87`
- [passage](https://tsep-protocol.org/en/updates/captures/48b084201543c4b6afd4ab2b7969e92d24ad51f88686821c70da9807e1035d3a.gz) · SHA-256: `48b084201543c4b6afd4ab2b7969e92d24ad51f88686821c70da9807e1035d3a`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## OAI-AdsBot

[oai-adsbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#oai-adsbot)

Checks the safety and relevance of pages submitted as ChatGPT ads.

No generative foundation-model training. The source does not precisely define how blocking affects ad approval.

User-Agent (documented example):

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
```

- [IP](https://openai.com/adsbot.json)

### OpenAI — documented

Document review: 2026-11-06

[Official source](https://developers.openai.com/api/docs/bots)

Captured: 2026-11-06T14:55:39+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/b63a02a9eb2ec848efc5dbeb16e63c04cf6d425fb79423bc534267a456c14cca.gz) · SHA-256: `b63a02a9eb2ec848efc5dbeb16e63c04cf6d425fb79423bc534267a456c14cca`
- [passage](https://tsep-protocol.org/en/updates/captures/6d11f6beab02ffe889ac8bfac323d0c69d4e7e5040b7bcc4406a2c49b0239818.gz) · SHA-256: `6d11f6beab02ffe889ac8bfac323d0c69d4e7e5040b7bcc4406a2c49b0239818`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## Google-CloudVertexBot

[google-cloudvertexbot](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#google-cloudvertexbot)

Controls site-owner-requested crawling for Vertex AI agents.

The documentation accepts Google-CloudVertexBot or Googlebot as a token; a dedicated rule avoids extending the choice to the Search crawler.

### Disallow this crawler

robots.txt

```text
User-agent: Google-CloudVertexBot
Disallow: /
```

User-Agent (documented example):

```text
Google-CloudVertexBot
```


### Google — documented

Document review: 2026-11-06

[Official source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

Captured: 2026-11-06T14:55:43+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/03b62207ae01bfa409db09177986f4df4b9574d74e6af9fbc0ad5a75bc00481e.gz) · SHA-256: `03b62207ae01bfa409db09177986f4df4b9574d74e6af9fbc0ad5a75bc00481e`
- [passage](https://tsep-protocol.org/en/updates/captures/d8c7efdef95daf256d2a6889b886443324243ef4d2068e587e0deebe69ef5030.gz) · SHA-256: `d8c7efdef95daf256d2a6889b886443324243ef4d2068e587e0deebe69ef5030`

[TS42](https://tsep-protocol.org/en/controls/TS42) [TS43](https://tsep-protocol.org/en/controls/TS43) 

## noarchive · Amazon

[amazon-noarchive](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#amazon-noarchive)

Asks Amazon not to use the page for training.

The page must remain accessible for the tag to be read. Other engines may also interpret the robots tag under their own rules.

### Documented page-level tag

HTML

```text
<meta name="robots" content="noarchive">
```


### Amazon — documented

Document review: 2026-11-06

[Official source](https://developer.amazon.com/docs/amazonbot/amazonbot.html)

Captured: 2026-11-06T09:56:24+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6.gz) · SHA-256: `ba11a19e5f9b6ae39ea2595b26c6e98fd84144f69cc1f7f443a29f4ac8cca1b6`
- [passage](https://tsep-protocol.org/en/updates/captures/215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4.gz) · SHA-256: `215416ad07464eaaecefb14a0782ff38c98aedd2037e7553e940f92a5f88e7a4`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## nosnippet · Apple

[apple-nosnippet](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#apple-nosnippet)

Excludes content from Apple’s AI broad world knowledge answers.

Does not by itself block Applebot or training. Traditional search results remain possible; snippets are suppressed.

### Applebot-specific tag

HTML

```text
<meta name="applebot" content="nosnippet">
```


### Apple — documented

Document review: 2026-11-06

[Official source](https://support.apple.com/en-us/119829)

Captured: 2026-11-05T17:05:18+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/30c9ef308e140287d2a6310879414df7f87a65314234b51d00bd971d6f4daad3.gz) · SHA-256: `30c9ef308e140287d2a6310879414df7f87a65314234b51d00bd971d6f4daad3`
- [passage](https://tsep-protocol.org/en/updates/captures/58cfc33eeebcef38f90dd04942c45862a15e4038451adfad5e4f7950709463cd.gz) · SHA-256: `58cfc33eeebcef38f90dd04942c45862a15e4038451adfad5e4f7950709463cd`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## Content-Usage

[content-usage](https://tsep-protocol.org/en/updates/ai-crawlers-and-robots-txt#content-usage)

The draft conveys a training opt-out preference associated with retrieved content.

Version -05 published 19 August 2026, expires 20 February 2027. AIPREF working-group draft, not an RFC. Preferences apply only to crawlable paths and are not retroactive. Presence does not establish provider support. Examples containing Allow: / must be reconciled with existing exclusions.

### Draft HTTP example

HTTP

```text
Content-Usage: train-ai=n
```

### Draft robots.txt example

robots.txt

```text
User-agent: *
Allow: /
Content-Usage: train-ai=n
```


### IETF AIPREF — documented

Document review: 2026-11-06

[Official source](https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.html)

Captured: 2026-11-06T14:56:27+00:00

- [raw](https://tsep-protocol.org/en/updates/captures/cf419c0e2ffd975e3f2f3f029e29efb3b9025a8feae6fb5fb4fed6c7998104a9.gz) · SHA-256: `cf419c0e2ffd975e3f2f3f029e29efb3b9025a8feae6fb5fb4fed6c7998104a9`
- [passage](https://tsep-protocol.org/en/updates/captures/54cd443ca845405c924c4d430c8da6ce82d98c483c168085b065bdfeca5a1cef.gz) · SHA-256: `54cd443ca845405c924c4d430c8da6ce82d98c483c168085b065bdfeca5a1cef`

[TS42](https://tsep-protocol.org/en/controls/TS42) 

## Revision history

- **2026-11-06.2** — Corrected Perplexity delay (up to 24 hours), clarified Google controls, readable Copy buttons, shared method and Markdown view. All 37 records retained.
- **2026-11-06** — Initial edition, expanded from 27 to 37 settings without changing its identifier. This ambiguity is recorded here; subsequent revisions carry a distinct suffix.