Site root only
Serve the file at the exact scheme, host, and port root as /robots.txt. A subdirectory copy does not control the rest of the host.
Robots.txt checker · page directives included
Check robots.txt and crawler access
Test the selected crawler against robots.txt, then inspect the page-level X-Robots-Tag and meta robots directives that a file-only checker can miss. The result proves policy behavior at snapshot time, not a crawler visit.
PAGE LIMIT2 MiBREDIRECTS5 maxTIMEOUT8 sec
⌁
Ready to test crawler access.
The report keeps robots.txt, HTTP headers, and page metadata as separate observations.
Policy authoring · path evaluation · browser-local
Build a reviewable robots.txt policy
Create crawler groups, allow and disallow paths, attach sitemap locations, and test the resulting policy before publishing it at the site root. Nothing is submitted or uploaded.
- FORMAT
- UTF-8 TEXT
- STANDARD
- RFC 9309
- OUTPUT
- TXT + JSON
TXT
Ready to build a policy.
The generated file, validation findings, and path test will appear here.
Publish the policy where crawlers can actually find it
Crawl is not indexing
A disallowed URL may still be known or shown without a snippet. Use a crawlable noindex response when the goal is search-result exclusion.
Verify the response
Check the deployed HTTP status, content type, redirects, and the exact target path. Robots policy is public and must never protect confidential content. Compare a baseline and current policy.