The 44 controls / TS07
TS07 · Indexing
Robots meta directives and X-Robots-Tag match the page objective.
Protocol wording · 0.1.0-draft.28 · Documentation view; no website assessment.
Control rule version : 0.1.0-draft.28
Reading guide: start with applicability and required evidence, then read expectations together with their limitations. Understand C, NC, NT and NA outcomes · Prepare a dossier
Expectations and evidence
- Required inputs (protocol identifiers)
- http, html, intent, robots
- When it applies
- Documents and resources with a declared indexing objective.
- How to check
- Review final headers, source HTML, access and required rendered states under A01–A03.
- Expectation
- Effective target-bot directives in final headers and HTML match that objective; unknown directives and any necessary rendered states are reviewed.
- Evidence to retain
- One directive or a compatible combination, with no accidental noindex.
- Non-applicability
- No resource subject to an indexing policy in scope.
- Limitations
- The engine must be allowed to crawl the resource to read the directive.
- Outcome clarification
- C requires all expectations across the declared scope. NC requires evidenced contradiction; otherwise NT. NA requires evidence of non-applicability.
- Reference scope
- Technical reference; the TSEP acceptance rule remains an editorial proposal. Sources checked on 2026-10-09; dated capture fingerprints recorded in docs/source-observations.md.
Atomic rules
TS07-A01 — Effective directives and objective
- Required inputs (protocol identifiers)
- http, html, intent
- When it applies
- Each document with a declared indexing objective, for an explicitly named crawler and context.
- How to check
- Inventory all final X-Robots-Tag headers and robots/target-crawler meta tags in source HTML, separate from redirect hops. Apply documented crawler semantics, including duplicates, scope, case, parameters and ignored rules. Optional source_reader=html-tree-directives-v1, declared before capture, constructs two complete static hypotheses without executing scripts; HTML robots/googlebot meta in head/body without foreign ancestors are combined separately with final headers. Ordinary template contents are retained but excluded from active directives. Optional prior presentation_reader=google-preview-intent-v1 compares minimum/maximum permission intervals for text, images and video under the preview-intent contract; extraction and states remain separate.
- Expectation
- Combined effects match the declared indexing objective and declared presentation/link restrictions. For Google: an applicable restriction prevails over permission, none includes noindex/nofollow; nofollow or a preview limit alone does not mean noindex. Missing directives are not failure when intent permits indexing. For this profile, two pass yield pass, two fail yield fail, every other pair stays inconclusive; never merge directives across alternative hypotheses or confuse them with observed rendered states. For this new profile, all possible values within the intended interval yield pass; all outside on one side yield fail; otherwise inconclusive. Independent contradiction prevails within a reading; existing source consensus and rendered priority remain.
- Evidence to retain
- Raw repeated headers, source HTML, token/crawler/effect table and dated technical reference. For non-HTML, the html-kind record documents why this surface does not apply, with Content-Type, without inventing HTML. This profile also retains both full meta inventories, contexts/namespaces, original tags/body-byte spans, templates, decoding, reported errors, versions and fingerprints in calculated observations linked to the report and compared during replay. The new profile additionally retains ordered HTTP/meta values, scopes, unknowns, dimension decisions and intervals in bound replayed observations.
- Assumptions
- Intent and crawler fixed before review; do not extrapolate Google rules to another engine. For indexifembedded or unavailable_after, include the required embedding context and date. Intent retains crawler as the exact HTTP User-Agent. Optional directive_crawler, frozen before collection and limited here to googlebot, selects directive semantics only for this client representation; it does not describe a response received by Googlebot. Without this field, historical behavior requires crawler=googlebot. Absent source_reader preserves the legacy reader; null/unknown or missing Node 24 never falls back. This profile requires explicit HTTP UTF-8 HTML, consumes one initial BOM and retains unreported repairs by the pinned parser. The bilingual source-directives contract specifies assumptions/bounds without asserting HTML validity. Bounds concern a permission cap, not actual display. Selection and bounds are prior; absent reader preserves historical behavior and no scalar is automatically migrated.
- Inconclusive outcome
- Missing objective, unknown syntax/scope, partial capture or missing context: inconclusive. A documented ignored directive is recorded, not automatically treated as an error. Present but unsupported, null or inconsistent directive_crawler: inconclusive; no favorable fallback. For this profile: incomplete reading, reported error, unsupported encoding, declared open/closed shadow, template for, foreign-context robots metadata, missing content/mixed roles or near form defined by the contract, unknown directive and exceeded bound remain inconclusive. Unreadable HTML never becomes header-only agreement. The preview profile retains unqualified image defaults, max-snippet:0 video interaction, unsupported syntax and exceeded bounds as unknown under the contract’s conservative intervals. Noindex does not give a presentation pass.
- Non-applicability
- No atomic exemption when the control applies.
- Limitations
- Assesses observed declarations, not ingestion or actual indexing. A01 alone does not cover access or rendering. Automation is semiAuto: the bounded interpreter does not cover every syntax, crawler or context; unsupported inputs remain inconclusive and require review. This methodological choice does not simulate Google. No rendering, access, review or TS10 outcome is inferred. The preview profile assesses neither Google display, image availability, data-nosnippet nor licensed, structured-data or AI usage; it produces no review.
TS07-A02 — Access to the directive
- Required inputs (protocol identifiers)
- http, robots, intent
- When it applies
- Every resource for which directive effects are assessed.
- How to check
- Compare effective robots policy and access conditions for the target crawler; distinguish the test client response, explicit blocking and unobserved crawler access. With explicit directive_crawler, require an explicit robots_policy whose product_token matches googlebot case-insensitively; select that group and never fall back to the browser agent group.
- Expectation
- No observed obstacle prevents reading the directive in the declared context. Robots disallow contradicts a strategy relying on reading noindex. An attributable HTTP 401/403 remains fail even if robots evidence is missing.
- Evidence to retain
- Robots capture and interpretation for URL/crawler, HTTP trace and access assumptions documented in intent.
- Assumptions
- Policy is interpreted by the shared TS04 reader. A new profile is declared in intent before capture. Without robots_policy, legacy inputs retain the fixed legacy-exact-v1 profile without implicit coverage extension. The client does not authenticate as Googlebot. Intent retains crawler as the exact HTTP User-Agent. Optional directive_crawler, frozen before collection and limited here to googlebot, selects directive semantics only for this client representation; it does not describe a response received by Googlebot. Without this field, historical robots profiles retain their behavior. Optional prior google-robotstxt-fields-v2 extends the closed registry of fields outside access scope using pinned google/robotstxt sources; old profiles remain unchanged with no automatic selection.
- Inconclusive outcome
- Unavailable or ambiguous policy, unresolved challenge, missing capture: inconclusive, never NA for missing access. The new path without a matching profile remains inconclusive. Attributable HTTP 401/403 denial retains precedence.
- Non-applicability
- No atomic exemption when the control applies.
- Limitations
- No evidence of an actual engine visit; TS04 and other access controls are not globally validated. Automation is semiAuto: the bounded interpreter does not cover every syntax, crawler or context; unsupported inputs remain inconclusive and require review. Registry fields and sitemap are recorded without assessing usage preferences, TS42 or indexing and never delimit groups. Other fields and tolerances remain unsupported. Full line numbers and values are retained; reason previews disclose truncation.
TS07-A03 — Required rendered states
- Required inputs (protocol identifiers)
- http, html, render, intent
- When it applies
- Pages whose scripts or states can affect directives, or whose stability is not established.
- How to check
- Fix required states, interactions and waiting conditions before capture. Compare final headers and source HTML with each DOM, declaring browser/version, enabled scripts, time, errors and context. Bind each capture to the source response and target. Do not assume the engine renders initially noindex HTML. The DOM reader is chosen before capture: structured-inventory-v2, or the fixed legacy serialization reader when the field is absent. No fallback after refusal. The complete structured inventory retains namespaces, positions, duplicates, serialization and navigation attribution records; no node is removed based on its presumed origin. The optional new source_reader affects only the A01 source prerequisite, never each DOM reader; priority for an attributable rendered contradiction remains unchanged. Optional prior presentation_reader=google-preview-intent-v1 compares minimum/maximum permission intervals for text, images and video under the preview-intent contract; extraction and states remain separate.
- Expectation
- All required states are documented and compatible with intent in the declared context. A contradiction in a complete, attributable state is fail even if other states are missing. Removing initial noindex with JavaScript does not yield pass: A03 remains inconclusive unless a required state supplies another contradiction; A01 retains its own source verdict. For this new profile, all possible values within the intended interval yield pass; all outside on one side yield fail; otherwise inconclusive. Independent contradiction prevails within a reading; existing source consensus and rendered priority remain.
- Evidence to retain
- Retained headers/source HTML, prior plan, dated raw DOMs with binding digest of the HTTP trace, final URL, context, browser/version, script state, interactions, waiting conditions and errors. Reasoned comparison and limitations; no simulated engine capture. The new profile additionally retains ordered HTTP/meta values, scopes, unknowns, dimension decisions and intervals in bound replayed observations.
- Assumptions
- Required states are listed before the test; local and engine rendering remain distinct. The structured reader interprets HTML meta children directly in the real document head; ambiguous placement or shadow remains inconclusive. It retains combination with the declared navigation headers. Intent retains crawler as the exact HTTP User-Agent. Optional directive_crawler, frozen before collection and limited here to googlebot, selects directive semantics only for this client representation; it does not describe a response received by Googlebot. Without this field, historical behavior requires crawler=googlebot. Bounds concern a permission cap, not actual display. Selection and bounds are prior; absent reader preserves historical behavior and no scalar is automatically migrated.
- Inconclusive outcome
- Missing required plan/capture, truncated state, script error, inconsistent source/target/context binding, invalid chronology, uninterpretable directive or late removal of initial noindex: inconclusive without an inferred pass. An established contradiction in another attributable state remains fail. For the structured reader: incomplete tree/transport, loading document, known API alteration, missing integrity history, separate source/DOM acquisition without same-navigation attribution or a different agent: inconclusive. A digest alone does not prove navigation. Present but unsupported, null or inconsistent directive_crawler: inconclusive; no favorable fallback. The preview profile retains unqualified image defaults, max-snippet:0 video interaction, unsupported syntax and exceeded bounds as unknown under the contract’s conservative intervals. Noindex does not give a presentation pass.
- Non-applicability
- Non-HTML or directive stability demonstrated by documented review; intent evidence references this justification and supporting records. No unsupported “no JS” assertion.
- Limitations
- semiAuto: the plan and sufficiency of observations require review. The interpreter compares supplied captures; it does not execute JavaScript or establish exhaustive states, TS25, engine rendering or indexing. Supplied attribution and APIs are not authenticated; authored synthetic records are explicitly labeled, never actual visits. Frames, template fragments and closed shadow roots are outside scope. Bounds and migration are documented in docs/structured-rendering.md. The preview profile assesses neither Google display, image availability, data-nosnippet nor licensed, structured-data or AI usage; it produces no review.