Robots.txt Generator & Tester
Build a correct robots.txt in seconds. Choose which search engines, AI bots and SEO crawlers may visit, block private folders, add your sitemap — then test any URL.
Test a URL
Your rules are built in your browser. “Import” only reads the public robots.txt of the site you enter.
How to use the Robots.txt Generator
Pick your rules
Start from a preset, then allow or block each crawler and folder.
Test URLs
Check whether Googlebot or any other bot can crawl a specific page.
Download
Download robots.txt and upload it to the root of your website.
Why use our Robots.txt Generator?
Smart presets
Allow all, WordPress, Laravel, or block AI training bots in one click.
25+ crawlers
Google, Bing, Yandex, GPTBot, ClaudeBot, CCBot, Ahrefs, Semrush and more.
Built-in tester
Test paths with Google’s matching rules, including * and $ wildcards.
Import live file
Load the current robots.txt from any website and edit it.
Sitemap lines
Add one or more sitemap URLs.
Safety warnings
Warns you before you accidentally block your whole site.
Frequently asked questions
Everything you need to know about the Robots.txt Generator.
Back to the toolWhat is robots.txt?
robots.txt is a text file at the root of your website (example.com/robots.txt) that tells crawlers which pages they may or may not visit.
Does robots.txt remove pages from Google?
No. Blocking a URL stops crawling, but the URL can still appear in results if other sites link to it. To remove a page, use a noindex meta tag and allow crawling.
How do I block AI bots like ChatGPT?
Use the “Block AI bots” preset. It disallows GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and other AI crawlers while keeping Google Search allowed. Well-behaved bots respect it.
Where do I upload robots.txt?
Upload it to the root folder of your domain (usually public_html or public) so it opens at https://yourdomain.com/robots.txt.
What does Crawl-delay do?
It asks bots to wait between requests. Bing and Yandex respect it; Google ignores it.
You may also like
Have a project in mind? Let’s build it together.
Get a free consultation and a fixed-price quote within 24 hours.