The 44 controls / TS04
TS04 · Robots and access
The effective robots policy matches declared intent.
Protocol wording · 0.1.0-draft.28 · Documentation view; no website assessment.
Control rule version : 0.1.0-draft.28
Reading guide: start with applicability and required evidence, then read expectations together with their limitations. Understand C, NC, NT and NA outcomes · Prepare a dossier
Expectations and evidence
- Required inputs (protocol identifiers)
- http, robots, intent
- When it applies
- Hosts and agents for which a crawl policy is assessed.
- How to check
- Declare origin, crawler, interpretation profile and resource/expected-policy matrix before collection; compare policy and access under A01–A02.
- Expectation
- Effective rules and resource access match the declared policy; documented intentional absence may be compatible, without universally requiring 200.
- Evidence to retain
- Prior plan, origin, product token and User-Agent, profile, dated raw robots, selected lines, required-resource responses and supplied reviews.
- Non-applicability
- No crawlable host in scope.
- Limitations
- robots.txt is a crawling protocol, not an access-control mechanism.
- Outcome clarification
- C requires all expectations across the declared scope. NC requires evidenced contradiction; otherwise NT. NA requires evidence of non-applicability.
- Reference scope
- Technical references observed October 10, 2026; captures/fingerprints in docs/source-observations.md. Acceptance remains a TSEP proposal.
Atomic rules
TS04-A01 — Policy per origin, crawler and resource
- Required inputs (protocol identifiers)
- http, robots, intent
- When it applies
- Each origin and path matrix whose crawl policy is assessed.
- How to check
- Freeze the profile and expected allowances/disallows before observation. Interpret robots status, groups, rules and redirects in the initial origin context; retain the lines determining each outcome.
- Expectation
- Every observed policy matches declared intent. Unexpected permission and unexpected disallow are contradictions. Intentional absence can be compatible under the profile; no universal 200 requirement.
- Evidence to retain
- Complete matrix, prior intent, context, full robots request/response, profile and reasoned per-resource result.
- Assumptions
- Choose the profile before measurement and keep it fixed for a campaign context; never select the favorable profile after observation. Distinguish RFC model, documented Google policy and unobserved actual cache. Optional directive_crawler=googlebot, frozen before capture, selects the Googlebot policy only on the representation received by crawler, the exact HTTP User-Agent. An explicit robots_policy with case-insensitive googlebot product_token is required, as for TS07. Without the field, historical behavior is preserved. Optional prior google-robotstxt-fields-v2 extends the closed registry of fields outside access scope using pinned google/robotstxt sources; old profiles remain unchanged with no automatic selection.
- Inconclusive outcome
- Absent profile/capture/timing, unsupported syntax or unresolved error/cache: inconclusive. An attributable contradiction on another resource remains fail. Null, unsupported or inconsistent directive_crawler makes policy interpretation inconclusive without falling back to the client group. An independent HTTP contradiction remains limited to the observed client.
- Non-applicability
- No atomic exemption when the control applies.
- Limitations
- semiAuto: matrix and profile suitability remain declared. The reference is bounded to one origin, documented profiles and supplied captures; no engine visit, ingestion or indexing is established. In explicit mode, every robots conclusion concerns the declared product on bytes received by the client, without proving the representation received by Googlebot. An HTTP status contrary to the plan (including 401/403/429/5xx when 200 is expected) remains fail for that client, never evidence of denial to Googlebot. Registry fields and sitemap are recorded without assessing usage preferences, TS42 or indexing and never delimit groups. Other fields and tolerances remain unsupported. Full line numbers and values are retained; reason previews disclose truncation.
TS04-A02 — Access to required resources
- Required inputs (protocol identifiers)
- http, robots, intent
- When it applies
- Resources explicitly required in the declared scope, including selected rendering resources.
- How to check
- Compare robots policy and HTTP trajectories with planned destinations/statuses, then review responses for obstacles. Bind each supplied review to bytes, dates, intent and context. Retain gaps separately.
- Expectation
- Every required resource and observed hop is allowed under the profile, reaches its planned response and has no obstacle according to supplied review. Robots blocking, a contradictory response or an evidenced obstacle gives fail even when other evidence is missing.
- Evidence to retain
- Resource plan, robots, complete HTTP traces, reasoned per-resource review with declared identity, date, provenance, assistance and binding digest.
- Assumptions
- A test client does not authenticate as Googlebot; 200 alone cannot establish absence of a challenge. Software does not produce the review or authenticate reviewer identity. Optional directive_crawler=googlebot, frozen before capture, selects the Googlebot policy only on the representation received by crawler, the exact HTTP User-Agent. An explicit robots_policy with case-insensitive googlebot product_token is required, as for TS07. Without the field, historical behavior is preserved. Optional prior google-robotstxt-fields-v2 extends the closed registry of fields outside access scope using pinned google/robotstxt sources; old profiles remain unchanged with no automatic selection.
- Inconclusive outcome
- Missing resource/review, unbound review, incomplete response, uncertainty or unsupported scope: inconclusive without erasing an established contradiction. Null, unsupported or inconsistent directive_crawler makes policy interpretation inconclusive without falling back to the client group. An independent HTTP contradiction remains limited to the observed client.
- Non-applicability
- No required resource in this scope, established by reasoned inventory review dated before the plan; an unjustified empty list is insufficient.
- Limitations
- Inventory sufficiency and human judgment are not fully automated. The reference covers one origin and attributable trajectories; it does not execute rendering or establish actual engine access. In explicit mode, every robots conclusion concerns the declared product on bytes received by the client, without proving the representation received by Googlebot. An HTTP status contrary to the plan (including 401/403/429/5xx when 200 is expected) remains fail for that client, never evidence of denial to Googlebot. Registry fields and sitemap are recorded without assessing usage preferences, TS42 or indexing and never delimit groups. Other fields and tolerances remain unsupported. Full line numbers and values are retained; reason previews disclose truncation.