Search · agents · training · archives
Crawler policy by purpose
Classify the behavior, verify the operator, model the cost and benefit, then express a robots policy. Identity alone never tells you whether a crawler visited your site or deserves access.
Check a robots policyBuilds a public search index
Usually allow when organic discovery is desired
Search provider documentation + verified identity
Retrieves or indexes content for answer/search experiences
Decide based on discovery value and content terms
Operator purpose and current crawler controls
Fetches at a user's request
Consider user benefit, rate, and authenticated identity
Request signature or documented user agent
Collects content for model development
Make a separate licensing and policy decision
Training purpose must not be inferred from generic AI labels
Preserves public web snapshots
Balance historical preservation with content constraints
Archive operator policy
Measures sites for webmaster tools
Allow only if the service is useful and well-behaved
Operator, rate, and declared purpose
The analyzer fetches robots.txt once, evaluates each operator-documented token against the requested path, and retains whether the result came from a specific rule, wildcard rule, default behavior, missing file, or unavailable fetch. It never sends a request as the named crawler. First-party logs and provider verification are still required for site-specific visit and identity evidence.