AI crawlers retrieve pages for search, training or user requests. These uses require different controls. Find documented rules, their effects and official evidence below. Every record is on this page.
Opt-out removes the site from ChatGPT search answers, although navigational links may remain.
Does not configure GPTBot. Allowing search also requires working network access.
Syntax, timing and identification +
Disallow this crawler
User-agent: OAI-SearchBot
Disallow: /
robots.txt
Stated delay
About 24 hours for search systems.
Documented User-Agent (example, versions may vary)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Blocking Applebot can remove presence in Apple search.
For training only, use Applebot-Extended; for AI broad world knowledge answers, use nosnippet. Without Applebot rules, Apple uses Googlebot rules if present.
Expresses a Bytespider crawl opt-out under ByteDance documentation.
Documentation alone does not establish observed compliance or identify all group traffic.
Syntax, timing and identification +
Disallow this crawler
User-agent: Bytespider
Disallow: /
robots.txt
Documented User-Agent (example, versions may vary)
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
Documented User-Agent (example, versions may vary)
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
Documented User-Agent (example, versions may vary)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
No dedicated robots.txt rule targets only AI Overviews or AI Mode: blocking Googlebot affects Google Search as a whole. Use nosnippet, data-nosnippet, max-snippet or noindex to limit display according to the intended scope; the Search Console setting can exclude generative features. A URL blocked from crawling may remain known: robots.txt does not replace noindex.
Syntax, timing and identification +
Disallow this crawler
User-agent: Googlebot
Disallow: /
robots.txt
Documented User-Agent (example, versions may vary)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Expresses a generative-training opt-out to providers implementing the signal.
User-agent: * does not express this preference. Without a dedicated group, no shared preference is stated. Provider-specific rules and private agreements may take precedence. The token does not block HTTP requests.
Individual IETF submission, version -00 dated 30 October 2026; expires 3 May 2027. Not an adopted standard.
Document reviewSource and capture · __usage_ai-training · Google
Google plans to map a shared opt-out to a Google-Extended disallow when no explicit Google-Extended rule applies. That mapping would also affect grounding in Gemini Apps and Grounding with Google Search on Vertex AI. An explicit rule takes precedence.
An explicit Google-Extended rule retains priority.
Microsoft · BingAnnounced
· Bing starts applying the new training and grounding preferences as it crawls.
Document reviewSource and capture · __usage_ai-training · Microsoft · Bing
Sites already using NoArchive or NoCache on December 15 can set a temporary grounding preference in Bing Webmaster Tools. For training, the page-level preference will take precedence over robots.txt.
Current controls remain; new-signal precedence is unspecified.
AnthropicRolling out
Since Oct 30, 2026 · gradual rollout
Document reviewSource and capture · __usage_ai-training · Anthropic
Anthropic says recognition has begun. A rule naming ClaudeBot takes precedence for the path concerned. The provider describes a gradual rollout across its services.
Disallow opts out of grounding. DisallowExtended limits it to Microsoft-operated experiences, excluding Microsoft grounding services for third parties.
Page-level only, never in robots.txt. Independent of training, with no change to eligibility for traditional Bing results.
Application announced from 1 February 2027, as pages are crawled.
Bing’s legacy AI preferences should be replaced before 1 March 2027.
For eligible sites: temporary grounding choice from 15 December 2026; new rules read from 1 February; temporary choice ends at recrawl or on 30 September 2027.
Migration: use the ai-training and ai-grounding records.
Syntax, timing and identification +
No syntax is presented as a guaranteed block for this record.
Declares search, ai-input and ai-train preferences. The documentation also presents an experimental use field.
use=immediate: interact without retention; reference: index, excerpt and link; full: summarize and reproduce. The page is dated 2 November; this does not establish the test start date. Neither network blocking nor universal compliance is guaranteed. Examples containing Allow: / must be reconciled with existing exclusions.
The draft conveys a training opt-out preference associated with retrieved content.
Version -05 published 19 August 2026, expires 20 February 2027. AIPREF working-group draft, not an RFC. Preferences apply only to crawlable paths and are not retroactive. Presence does not establish provider support. Examples containing Allow: / must be reconciled with existing exclusions.
Each link leads to the relevant setting. This matrix classifies documented uses; it measures neither compliance nor universal compatibility. A dash means “no record in this register”.
Settings by provider and use, with links to evidence
Can you opt out of training while remaining visible in search?
Yes, where the provider separates these uses. OpenAI uses GPTBot for training and OAI-SearchBot for search. Google-Extended does not remove a site from Google Search. A blanket rule disallowing every crawler would have a very different scope.
No. It expresses a rule for the client to interpret. Some user-triggered accesses may not apply it. Effective refusal must be checked in server responses and logs; a claimed User-Agent identity alone cannot establish who made a request.
A capture retains the received document. Its hash allows its bytes to be verified; an independent timestamp establishes that the manifest existed no later than the signed time. Neither proves the original publication date, the truth of an announcement or actual crawler compliance.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
This capture covers the OpenAI news feed. The dossier separately records the article passage used for the timeline. View the evidence
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
The shared signal is an individual IETF submission. The AIPREF working group continues its work. Google relates its proposal to the European General-Purpose AI Code of Practice.
AI-assisted documentary readings based on archived official sources. “Documented” describes what a provider publishes, not observed behavior. A reading is withdrawn if a supporting source passage changes.
Review dates, capture dates and effective dates are distinct. An omitted effective date or delay means the supporting source does not establish it; it does not mean immediate application or no delay.
Examples are targeted: integrate them with existing rules and check their scope before publishing. A User-Agent can be spoofed; its name alone cannot attribute a request.
SHA-256 hashes apply to decompressed captures. Compared text is a normalized extraction. Independent timestamps establish the existence of a manifest, not the truth of a document or crawler compliance.
Human review of this edition: pending. The signature on the 30 October dossier does not cover this register.
Revision history
2026-11-06.2 · Corrected Perplexity delay (up to 24 hours), clarified Google controls, readable Copy buttons, shared method and Markdown view. All 37 records retained.
2026-11-06 · Initial edition, expanded from 27 to 37 settings without changing its identifier. This ambiguity is recorded here; subsequent revisions carry a distinct suffix.
Open data
Cite. Reuse.
Edikka — TSEP. « AI crawlers: list, uses and robots.txt rules ». Edition 2026-11-06.2. tsep-ai-crawlers.
