The Robots.Txt File Checklist: What to Verify Before You Publish

Run through this before you consider robots.txt file actually done:

1. Never use robots.txt to try to keep a page out of the index — it only blocks crawling, not indexing; a blocked page with external links pointing to it can still show up in search results with no description.
2. Use noindex meta tags (which require the page to be crawlable) for pages you actually want excluded from search results.
3. Point to your sitemap with a `Sitemap:` line at the bottom of the file so crawlers find it without needing a separate submission.
4. Test changes in Google Search Console's robots.txt tester before deploying — the file affects every crawler simultaneously and mistakes are immediately live.
5. Double-check this after any site migration or major redesign, not just at launch.

None of these individually take long. The value is in doing all of them, every time, instead of remembering most of them most of the time.

Pair this with the structured data (schema markup) checklist if you're doing a fuller pre-launch pass.

Robots.txt Generator runs through most of this automatically.

Everything in this guide runs directly in your browser — no signup, no install.