Updated guidance on robots.txt specifications
Evidence
- Google Crawling · documentation · 2026-10-23 Updated guidance on robots.txt specifications
- Official publication
- First detected
- 2026-11-05
- Last seen in the source
- 2026-11-12
- Topic
- Crawling and indexing
- Status
- Documentation updated
Added during the historical import.
Shortened excerpt · normalized text, not verbatim.
What: Updated the how Google interprets the robots.txt specification to add recommendation for percent-encoding in rule paths.…
Updated guidance on robots.txt specifications
Captures were made on the dates below, sometimes after first detection. Hashes apply to decompressed files. Compared text is a normalized extraction; the source capture retains the received bytes.
First capture · · Passage last seen on 2026-11-12
- Source capture · Download (gzip) · SHA-256
0b016d23d83635edc32fc8a7da77a0dab84eb1bcca35f5737c55a2a6b0b8e653- Compared text · Download (gzip) · SHA-256
c7bd24d3b0eeab2020ef66c1df8e0552c8f2a67b88ab29c336121cad7f8917d5
Existence timestamped by DigiCert on · Verify timestamp