robots.txt is a plain text file at the root of your website that tells crawlers which parts of the site they are allowed to visit. It is simple, powerful and easy to get wrong – a single line can stop search engines crawling your entire site.
This guide covers how it works, how to write it safely and how to handle the growing number of AI crawlers.
Where robots.txt lives
The file must be at the root of each host:
https://example.com/robots.txt
It only applies to that exact host and protocol. blog.example.com needs its own file, and so does http:// if it serves content (it should redirect to HTTPS anyway).
If there is no robots.txt (the server returns 404), crawlers assume they may crawl everything.
The basic syntax
User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/public-help/
Sitemap: https://example.com/sitemap.xml
- User-agent names the crawler a group of rules applies to.
*means all crawlers. - Disallow lists a path prefix the crawler should not fetch.
- Allow creates an exception inside a disallowed area.
- Sitemap gives the full URL of an XML sitemap. It can appear anywhere in the file and applies to all crawlers.
Lines starting with # are comments.
Wildcards
Google, Bing and most major crawlers support two special characters:
*matches any sequence of characters:Disallow: /*?sort=blocks any URL containing?sort=$marks the end of the URL:Disallow: /*.pdf$blocks URLs ending in.pdf
How rules are matched
robots.txt is now an official internet standard (RFC 9309, published in 2022), based on how Google has long handled it:
- A crawler uses the group whose
User-agentmatches it most specifically. Googlebot follows aUser-agent: Googlebotgroup if one exists, and ignores the*group. - Within that group, the most specific (longest) matching rule wins.
- If an Allow and a Disallow rule are equally specific, Allow wins.
Example:
User-agent: *
Disallow: /shop/
Allow: /shop/sale/
/shop/sale/shoes is allowed (the Allow rule is longer), while /shop/new/shoes is blocked.
Fetch and validate robots.txt, and test whether any URL is blocked for Googlebot or AI bots.
What robots.txt does – and does not do
It controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it, usually shown without a description. To keep a page out of search results, allow crawling and use a noindex robots meta tag or X-Robots-Tag header instead.
It is public. Anyone can read your robots.txt. Never use it to hide private areas – it effectively advertises them. Protect private content with authentication.
It is voluntary. Reputable crawlers follow it; malicious bots ignore it.
Safe examples
Allow everything (and point to the sitemap)
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/wp-sitemap.xml
Online shop
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?*sort=
Disallow: /*?*filter=
Sitemap: https://example.com/sitemap.xml
Blocking sorting and filtering parameters prevents crawlers wasting time on thousands of near-duplicate URLs.
Staging site (block everything)
User-agent: *
Disallow: /
Use this only on staging or development sites – and better still, protect them with a password. Forgetting to remove it at launch is one of the most common causes of a site vanishing from search.
Build a robots.txt file with sitemap, blocked folders and AI crawler controls.
Handling AI crawlers
AI companies run several kinds of crawlers, and it pays to treat them differently:
| Purpose | Examples | Effect of blocking |
|---|---|---|
| Training – collecting content to train models | GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google’s Gemini training control), CCBot (Common Crawl), Applebot-Extended | Your content is not used for future training. No effect on search visibility. |
| AI search – fetching pages to answer and cite in AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages may not appear as sources in those AI search engines. |
| User-requested fetches – when a person asks an assistant to read a page | ChatGPT-User, Claude-User, Perplexity-User | The assistant may be unable to read your page when users ask. |
Google-Extended is a control token rather than a separate crawler: blocking it stops content being used for Gemini model training but does not affect Google Search or AI Overviews, which rely on Googlebot.
A common middle-ground configuration blocks training while staying visible in AI search:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Check the providers’ documentation periodically, since crawler names and purposes change.
Check if ChatGPT, Claude, Perplexity and Google AI can access, read and quote your page.
Common mistakes
- Leaving
Disallow: /from staging. Always check robots.txt after a launch or migration. - Blocking CSS and JavaScript. Google needs them to render pages; blocking them can make pages look broken to Googlebot.
- Using robots.txt to remove pages from Google. Blocked pages cannot be seen to be noindexed. Use noindex instead.
- Blocking pages that have a canonical or redirect you want search engines to see.
- Typos and wrong case. Paths are case-sensitive:
/Admin/and/admin/are different. - A huge file. Google only reads the first 500 KiB.
- Serving robots.txt with errors. If robots.txt returns a 5xx error, Google may temporarily stop crawling the site.
Testing changes
Before you publish a new robots.txt, test your most important URLs against it – the homepage, key categories and product pages, and your CSS and JavaScript files. Search Console’s robots.txt report shows the version Google last fetched and any problems it found.
robots.txt vs noindex vs password protection
These three tools solve different problems, and mixing them up is a common source of indexing trouble:
| Goal | Right tool | Why |
|---|---|---|
| Save crawl time on low-value URLs (filters, internal search) | robots.txt Disallow | Stops crawlers fetching them at all |
| Keep a page out of search results | noindex meta tag or X-Robots-Tag | Crawlers must be able to fetch the page to see it |
| Keep content private | Password or login | Only authentication actually prevents access |
| Remove a page from Google quickly | noindex plus the Removals tool in Search Console | The tool hides it temporarily while noindex takes effect |
Handling parameters and faceted navigation
E-commerce and listing sites often generate URLs like these for every combination of filters:
/shoes?colour=red
/shoes?colour=red&size=8
/shoes?colour=red&size=8&sort=price
If these pages add no unique value for searchers, block the parameters that create endless combinations, such as sorting:
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*sessionid=
Be careful not to block filter pages that you do want to rank – for example, a “red running shoes” page with real search demand should be a crawlable, indexable URL with its own content.
Reading robots.txt for any site
Looking at competitors’ robots.txt files can be instructive. You can see which sections they keep crawlers out of, where their sitemaps live and how they handle AI crawlers. Because the file is public, it is also the first thing to check when a site you manage suddenly loses visibility.
Testing and monitoring
robots.txt changes are high-stakes, so treat them like code:
- Keep a copy of the previous version before editing
- Test important URLs – the homepage, key categories and products, CSS and JavaScript files – against the new rules before publishing
- After publishing, check that the file returns a 200 status and plain text, not an HTML error page
- Monitor Search Console’s robots.txt report and Page indexing report over the following days
Some content management systems and SEO plugins generate robots.txt dynamically. If you upload a physical robots.txt file, it usually overrides the virtual one – make sure you are editing the version that is actually served.
Key takeaways
- robots.txt controls crawling, not indexing – use noindex to keep pages out of search results.
- The most specific matching rule wins, and Allow beats Disallow in a tie.
- Never block CSS or JavaScript files that pages need to render.
- Separate AI training crawlers from AI search crawlers; blocking training does not affect search.
- Always check robots.txt after launches and migrations for leftover Disallow: / rules.
- Test important URLs against any change before you publish it.
Frequently asked questions
Do I need a robots.txt file?
Not strictly. Without one, crawlers access everything. But a robots.txt is the standard place to list your sitemap and keep crawlers out of low-value areas, so most sites should have one.
Does Google support crawl-delay?
No. Googlebot ignores Crawl-delay. Bing and some other crawlers respect it.
How long until changes take effect?
Google generally caches robots.txt for up to 24 hours, so changes can take a day to be picked up.
Can I block a specific page?
Yes – Disallow: /private-page – but remember this blocks crawling, not indexing. Use noindex if the aim is to keep it out of search results.
Check your own site
Run a free audit with 55+ checks and get a fix for every issue.