Robots meta · X-Robots-Tag · snippet controls

Build and review page-level crawler directives

Create robots meta tags and X-Robots-Tag headers for HTML pages or non-HTML files. OpenWebmaster also checks pasted directives for obvious conflicts, unsupported tokens, and preview-control mistakes.

DIRECTIVESLOCALOUTPUTSHTML + SERVERPROOFNOT IMPLIED

DIRECTIVE BUILDER

Choose the exact crawler instructions

Leave everything off for the normal crawlable-page default. Select restrictive directives only when the page or file should be handled differently.

Target crawlerCustom crawlernone Equivalent to noindex, nofollownoindex Do not show this page in search resultsnofollow Do not follow links on this pagenoarchive Do not show a cached copynosnippet Do not show text snippetsnotranslate Do not offer translated result copynoimageindex Do not index images on this pagemax-snippetmax-image-previewmax-video-preview

PASTED REVIEW

Check existing meta tags or headers

Paste page source, selected head markup, response headers, or implementation notes. The checker only reviews explicit robots meta and X-Robots-Tag lines it can see.

Existing robots directives 0 / 40,000 characters

Boundary: this builds and reviews directives; it does not prove crawlers can fetch the URL or that search engines have processed it.

ROB

Ready for your directive policy.

No sample noindex result, crawler-removal promise, AI-training guarantee, ranking impact, traffic claim, or manufactured robots score appears before you provide settings or source.

IMPLEMENTATION NOTES

Use crawler directives where they can be discovered

Do not block discovery by mistake

A crawler has to fetch a page or file to see a meta robots tag or X-Robots-Tag header. Robots.txt blocking and noindex solve different problems.

Prefer page-specific intent

Use noindex for URLs that should stay accessible but out of search. Keep important pages crawlable and avoid adding restrictive directives site-wide by accident.

Check the live response

After publishing, inspect the final HTML and headers with the URL indexability checker and HTTP header checker, then monitor Search Console or Bing Webmaster Tools separately.