Technical SEOEvidence Protocol
The register · SEO and GEO
Indexing, snippets, crawling. Compare six engines’ rules, their limits and the sources that document them.
The comparison
Rules are not universal. Start with the relevant engine; open a record for its example and caveats.
No matching rule. Try another term.
Scroll within the table: engine names and rule names stay visible.
| Mechanism | Bing | Yandex | Naver | Baidu | Applebot | |
|---|---|---|---|---|---|---|
noindexExclude a page from the index | DocumentedS12 | DocumentedS08 | DocumentedS17 | DocumentedS15 | Unconfirmed | DocumentedS01 |
nofollowAsk the crawler not to follow links on the page | DocumentedS12 | DocumentedS08 | DocumentedS17 | DocumentedS15 | Unconfirmed | DocumentedS01 |
nosnippetSuppress the page snippet | DocumentedS12 | DocumentedS05 | Unconfirmed | Unconfirmed | Unconfirmed | DocumentedS01 |
max-snippetLimit snippet length | DocumentedS12 | DocumentedS07 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
max-image-previewSet the maximum image preview size | DocumentedS12 | DocumentedS07 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
max-video-previewLimit video preview length | DocumentedS12 | DocumentedS07 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
data-nosnippetExclude a passage from the snippet | DocumentedS12 | DocumentedS05 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
noimageindexPrevent images from being indexed from this page | DocumentedS12 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
indexifembeddedIndex embedded content despite noindex | DocumentedS12 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
unavailable_afterSpecify when a page should stop appearing in results | DocumentedS12 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
notranslateDisable translation in results | DocumentedS12 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
noarchiveControl the cached copy or specific AI uses depending on the engine | UnusedS12 | Announced transitionS03 | DocumentedS17 | Unconfirmed | Unconfirmed | Unconfirmed |
nocacheLegacy cache and AI-use control | UnusedS12 | Announced transitionS03 | Unconfirmed | Unconfirmed | Unconfirmed | Unconfirmed |
X-Robots-TagSend directives through an HTTP header | DocumentedS12 | DocumentedS07 | DocumentedS17 | Unconfirmed | Unconfirmed | DocumentedS01 |
rel=canonicalIdentify the preferred URL among duplicates | DocumentedS10 | DocumentedS04 | DocumentedS18 | DocumentedS15 | DocumentedS02 | Unconfirmed |
hreflangAssociate language or region variants | DocumentedS13 | Unconfirmed | DocumentedS20 | Unconfirmed | Unconfirmed | Unconfirmed |
DisallowRestrict crawling of a path | DocumentedS11 | DocumentedS08 | DocumentedS21 | DocumentedS16 | Unconfirmed | DocumentedS01 |
AllowAllow crawling of a path | DocumentedS11 | DocumentedS08 | DocumentedS21 | DocumentedS16 | Unconfirmed | DocumentedS01 |
SitemapProvide the absolute URL of a sitemap | DocumentedS11 | DocumentedS09 | DocumentedS21 | DocumentedS16 | Unconfirmed | Unconfirmed |
Crawl-delayAsk a crawler to space out requests | UnsupportedS11 | DocumentedS06 | UnusedS19 | Unconfirmed | Unconfirmed | UnsupportedS01 |
Clean-paramIdentify URL parameters that do not change the content | UnsupportedS11 | Unconfirmed | DocumentedS21 | Unconfirmed | Unconfirmed | Unconfirmed |
IndexNowNotify engines of added, updated or deleted URLs via an API. Each engine decides whether to index them. | Not listedS14 | DocumentedS14 | DocumentedS14 | DocumentedS14 | Not listedS14 | Not listedS14 |
noindexExclude a page from the index
Exclude a resource from results.
The crawler must read the rule. Disallow can prevent discovery; noindex does not secure confidential content.
<meta name="robots" content="noindex">nofollowAsk the crawler not to follow links on the page
Indicate that links on this page should not be followed.
Does not deindex the page or destinations. A crawler may discover URLs through other links.
<meta name="robots" content="nofollow">nosnippetSuppress the page snippet
Exclude text snippets and some display or AI-response uses.
Scope varies by engine. Not a general AI-training opt-out; consult the AI crawler register for that use.
<meta name="robots" content="nosnippet">Also covers Discover, AI Overviews and AI Mode; prevents direct input to those two AI features. A static thumbnail may remain.
Excludes descriptions and web answers, and content use as context for Apple AI responses; URL discovery remains possible.
max-snippetLimit snippet length
Set a maximum snippet length in characters.
0 suppresses the snippet; -1 lets the engine choose. Guarantees neither a snippet nor an AI citation.
<meta name="robots" content="max-snippet:160">max-image-previewSet the maximum image preview size
Define the maximum image-preview size.
Values: none, standard, large. Permission does not require an image display. This is not noimageindex.
<meta name="robots" content="max-image-preview:large">max-video-previewLimit video preview length
Limit video previews to a number of seconds.
0 allows at most a static image; -1 imposes no limit. The video may remain indexable.
<meta name="robots" content="max-video-preview:10">data-nosnippetExclude a passage from the snippet
Exclude a passage from snippets without removing the whole page.
Content remains accessible. Google documents span, div and section; Bing accepts any HTML element.
<div data-nosnippet>Passage exclu / Excluded passage</div>Any HTML element. Content remains indexable and available for ranking, but is excluded from snippets and AI summaries.
noimageindexPrevent images from being indexed from this page
Request that images on the page not be indexed.
Documented here for Google. Distinct from thumbnail size or blocking access to an image file.
<meta name="googlebot" content="noimageindex">indexifembeddedIndex embedded content despite noindex
Allow content to be indexed when embedded elsewhere.
For Google, only takes effect alongside noindex; do not assume all engines support this exception.
<meta name="googlebot" content="noindex, indexifembedded">unavailable_afterSpecify when a page should stop appearing in results
Schedule the end of a page’s appearance at a specified date.
Google must read the rule. Use a valid date and explicit time zone; adapt the example to the intended expiry.
<meta name="googlebot" content="unavailable_after: 2027-12-31T23:59:59+00:00">notranslateDisable translation in results
Ask Google not to offer result translation.
Does not define page language and does not replace hreflang.
<meta name="googlebot" content="notranslate">noarchiveControl the cached copy or specific AI uses depending on the engine
Legacy cached-copy control whose scope depends on the engine.
No effect in Google Search. In Bing, do not confuse legacy AI-use controls with deindexing; migration announced for 2027.
<meta name="robots" content="noarchive">For training and grounding: new preferences honoured from 2027-02-01; migrate before 2027-03-01. Legacy rules are disregarded when new preferences are detected on the site.
nocacheLegacy cache and AI-use control
Legacy Bing preference related to caching and AI uses.
Not an HTTP browser-cache control. Google does not use it. Consult the Bing transition schedule before changing it.
<meta name="bingbot" content="nocache">For training and grounding: new preferences honoured from 2027-02-01; migrate before 2027-03-01. Legacy rules are disregarded when new preferences are detected on the site.
X-Robots-TagSend directives through an HTTP header
Send directives in the HTTP response, including for a PDF.
A delivery mechanism, not a standalone rule. Check each value and crawler target; an inaccessible rule cannot be read.
X-Robots-Tag: noindexrel=canonicalIdentify the preferred URL among duplicates
Suggest the preferred URL for identical or similar content.
A signal, not a redirect or indexing prohibition. Google may choose a different canonical from the declared one.
<link rel="canonical" href="https://example.com/page/">Bing recommends canonical tags when multiple versions must remain accessible. Source: official article dated 19 December 2025; a preference signal, not a redirect.
Historical official guide: one tag in head; the target must not be an error or blocked by robots.txt.
hreflangAssociate language or region variants
Connect language or regional versions of the same content.
For Google, reciprocal and self-referencing links. Does not replace canonical or actual translation.
<link rel="alternate" hreflang="fr" href="https://example.com/fr/">
<link rel="alternate" hreflang="en" href="https://example.com/en/">DisallowRestrict crawling of a path
Ask the crawler not to crawl a path.
May not prevent the URL appearing in results and does not secure a directory. A noindex rule behind the block may be unreadable.
User-agent: Googlebot
Disallow: /internal-search/AllowAllow crawling of a path
Allow a more specific path within a restricted area.
Precedence depends on the crawler and User-agent group. This Google example uses the most specific rule; equal matches prefer the least restrictive.
User-agent: Googlebot
Disallow: /assets/
Allow: /assets/public/SitemapProvide the absolute URL of a sitemap
Tell crawlers the absolute location of a sitemap.
Helps discover URLs; does not guarantee indexing or override access restrictions.
Sitemap: https://example.com/sitemap.xmlCrawl-delayAsk a crawler to space out requests
Adjust crawl pacing when the crawler supports it.
Documented by Bing; unsupported by Google and Applebot, retired by Yandex. Avoid treating it as universal.
User-agent: bingbot
Crawl-delay: 5Clean-paramIdentify URL parameters that do not change the content
Tell Yandex which parameters do not change content.
Must not hide a useful content variant. Yandex-specific in this edition’s sources.
User-agent: Yandex
Clean-param: ref /catalog/IndexNowNotify engines of added, updated or deleted URLs via an API. Each engine decides whether to index them.
Notify a URL addition, change or deletion.
Not a robots.txt directive. Accepted notification guarantees neither immediate crawling nor indexing; an ownership key is required.
https://api.indexnow.org/indexnow?url=https%3A%2F%2Fexample.com%2Fpage%2F&key=REPLACE_WITH_SITE_KEYBefore changing a rule
Limiting crawling, removing a page, reducing a snippet and declaring an AI-use preference are different decisions.
Check robots.txt and the HTTP response before tags. A noindex rule on a blocked page may never be discovered.
Retain the served response, target crawler and date. Then compare crawl and indexing observations; changes have no guaranteed instant effect.
Search and artificial intelligence
Google documents AI Overviews and AI Mode scope for nosnippet and max-snippet. Bing documents AI-summary exclusion with data-nosnippet. Apple connects nosnippet to AI-answer context. These rules do not replace operators’ training preferences.
Compare AI crawlers and controlsBing transition: the complete timelineMethod and evidence
Documentary comparison of six engines, based on official pages archived for this edition. “Documented” describes a declared rule, not behaviour observed on a test site.
“Unconfirmed” means retained sources do not establish support. It does not mean “unsupported”. For IndexNow, “Not listed” refers only to published endpoints.
Examples are illustrative and scoped. Precedence is not universal: Google generally applies the most restrictive meta rule; Yandex documents different precedence for permitting indexing rules. Crawlers must be able to access tags and headers.
The verification date refers to the source capture, not the rule’s launch. Timelines contain only explicitly documented dates. Readings and translations are AI-assisted; human review is stated separately.
Each capture contains received bytes compressed with gzip. SHA-256 applies to the decompressed document. Independent manifest timestamping proves captures existed no later than the signed time, not the truth of claims or observed crawler behaviour.
Human review FR/EN : Bertrand Morel · 2026-11-07
Independent timestamp
21 captures bound to the manifest signed on .
Download the verification bundleThe ZIP includes the manifest, signed token, certificates and OpenSSL command. It proves that captures existed, not that a crawler follows a rule.
2026-11-07.1 · FR/EN review confirmed by Bertrand Morel. Reservation on point 30: the 2009 Bing reference announced future support. At his request, it is replaced by the official 19 December 2025 article recommending canonical for duplicates. Documentary correction made by the assistant; new capture and newly timestamped manifest. Other captures retain their 6 November dates.
2026-11-06.1 · 22 mechanisms; examples, limits, engine differences, downloadable captures and exports. Correction: Applebot does not follow Crawl-delay. Announced transitions distinguished from applied rules.
2026-11-05 · Initial matrix of 16 mechanisms for six engines.
Open data
Edikka — TSEP. « SEO and GEO directive matrix ». Edition 2026-11-07.1. https://tsep-protocol.org/en/updates/seo-geo-directive-matrix. CC BY 4.0.
CSV contains one row per mechanism and engine. JSON retains nuances, examples, dates and complete hashes; historical AI data remains available in ai_rows.
CC BY 4.0 · TSEP factual compilation and editorial classifications. Source documents retain their original terms.
All TSEP registers