Robots.txt Generator
Build a robots.txt file with sitemap, blocked folders and AI crawler controls.
About the Robots.txt Generator
A robots.txt file tells crawlers which parts of your site they may visit. It is the right place to keep bots out of admin areas, internal search results, carts and endless filter combinations – and to point every crawler to your XML sitemap.
This generator builds a clean, valid file from a few choices, including presets for common platforms and separate controls for AI training crawlers (which collect content to train models) and AI search crawlers (which fetch pages to answer and cite in AI search results). You can block training while still appearing in AI search. Upload the result to the root of your domain as /robots.txt.
Frequently asked questions
Can robots.txt hide private pages?
No. The file is public and well-behaved bots merely choose to obey it. Protect private content with passwords or noindex, not robots.txt.
Will blocking AI training bots affect Google rankings?
No. Google-Extended only controls whether content is used for Gemini training; Googlebot and Google Search are unaffected.
Should I block CSS and JavaScript?
No. Google needs them to render pages properly. Blocking them can hurt how your pages are understood.