Robots.txt Tester
Check whether Googlebot, Bingbot, AI crawlers and others may crawl a URL — and see exactly which robots.txt rule decides it.
File check
robots.txt is fetched once by our server. Testing runs in your browser.
How to use the Robots.txt Tester
Load a robots.txt
From any website, or paste your own.
Enter URLs to test
One or many paths at once.
See the result
Allowed or blocked per bot, with the rule.
Why use our Robots.txt Tester?
15+ crawlers
Google, Bing, Yandex, GPTBot, ClaudeBot, CCBot and more.
Google’s rules
Longest match wins, * and $ wildcards supported.
Edit & test
Change the file and test before uploading.
Bulk URLs
Test many URLs in one go.
Syntax check
Finds unknown lines and common mistakes.
Sitemaps
Lists sitemap URLs declared in the file.
Frequently asked questions
Everything you need to know about the Robots.txt Tester.
Back to the toolWhat is robots.txt?
A text file at the root of a website that tells search engine crawlers which pages they may or may not crawl.
Does blocking a page remove it from Google?
No. robots.txt stops crawling, not indexing. Use a noindex tag to keep a page out of search results.
How are conflicting rules decided?
Google uses the most specific (longest) matching rule. If Allow and Disallow are equally long, Allow wins.
Can I block AI crawlers?
Yes. Add groups for GPTBot, ClaudeBot, CCBot, Google-Extended and others with Disallow: /.
You may also like
Have a project in mind? Let’s build it together.
Get a free consultation and a fixed-price quote within 24 hours.