Robots.txt Tester
Fetch and validate robots.txt, and test whether any URL is blocked for Googlebot or AI bots.
About the Robots.txt Tester
robots.txt is a plain text file at the root of a site (/robots.txt) that tells crawlers which parts of the site they may visit. A single wrong line – such as Disallow: / left over from a staging site – can stop search engines crawling everything.
This tester fetches the live file, validates every line and tells you whether a given path is allowed for the crawler you choose, showing exactly which rule matched. It follows the official robots.txt standard (RFC 9309) that Google uses: the most specific (longest) rule wins, and Allow wins a tie. It also shows at a glance which AI crawlers – such as GPTBot, ClaudeBot and PerplexityBot – can access the path.
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag.
Should I block AI crawlers?
It is your choice. Blocking training bots (like GPTBot or Google-Extended) does not affect Google Search. Blocking search bots (like OAI-SearchBot or PerplexityBot) can stop your site appearing in those AI search results.
Where must robots.txt be placed?
At the root of each host and protocol: https://example.com/robots.txt. Subdomains such as blog.example.com need their own file.