Free tool
Tester
Free robots.txt Generator & Tester (with AI Crawlers)
Build a robots.txt that allows or blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, then test any path before you publish.
AI crawlers
Blocking a training crawler does not remove you from search, but blocking a search crawler can stop citations. Decide each one on purpose.
robots.txt
Tester
This tool runs entirely in your browser. Nothing you type is sent to a server.
How to use it
- Pick a preset, then switch individual crawlers between allow and block.
- Add any paths that no crawler should visit, and your sitemap URL.
- Test a path against a crawler to confirm the result before you publish.
- Save the file as robots.txt in the root of your domain.
What to keep in mind
- robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it.
- Not every crawler obeys robots.txt; it is a request, not access control.
- The tester follows the common longest-match rule, but check important changes in Search Console.
- User agent names change over time, so review the list against each provider’s documentation.
Frequently asked questions
Should I block AI crawlers?
It depends on your goals. Blocking training crawlers and allowing search crawlers is a common middle path, but blocking search crawlers can stop AI citations.
Does Google-Extended affect Google Search?
No. Google says it does not affect inclusion or ranking in Google Search; it controls use of content for Gemini training and grounding.
Where does robots.txt go?
At the root of each host, for example example.com/robots.txt. Subdomains need their own file.