Free tools / robots.txt tester
robots.txt tester and generator for AI crawlers
Our robots.txt tester reads your live robots.txt and shows, for each of 11 AI and search crawlers, whether it may read your site and which rule decides. Test any path for any crawler, then generate a corrected robots.txt to copy or download.
We read one file, /robots.txt, within eight seconds and up to 750 KB, following up to three redirects, each to a public HTTPS address. Path tests, pasted files and the generator run in this page.
Paste a robots.txt instead
The pasted file is read in this page and is not sent to our server.
Generate a robots.txt
A crawler named in its own group ignores the * group, so the draft repeats your private paths in each group that allows reading.
No email is needed; runs are limited per network and per day, and by the site's daily allowance.
We do not save your website, company name, page content or result. We keep daily totals and hashed network counters to enforce limits. A shared result stays in its link, not on our server.
What we check and why
Which group applies to each crawler
A crawler follows the group that names it and ignores the * group when one does. We show which group decides for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended and Bingbot, so you can see which line to change.
The rule that decides a path
Within a group the longest matching rule wins, and Allow wins a tie, as the robots.txt standard, RFC 9309, sets out. We match * and $ the same way, so a test here gives the answer a crawler that follows the standard gives.
Sitemap lines
Sitemap: lines tell crawlers where your list of pages is. We list them so you can check they point at your current sitemap.
A file you can publish
The generator writes one group for each crawler you choose and repeats your private paths in each group that allows reading. Review the draft, then publish it at the root of your domain as text/plain.
Each crawler's role and its operator's documentation are listed in our crawler identity inspector.
Questions
Does blocking GPTBot in robots.txt keep my site out of ChatGPT?
Not by itself. OpenAI's crawler documentation lists GPTBot for training-related crawling, OAI-SearchBot for ChatGPT search and ChatGPT-User for visits a person asks for. They are separate names, so a Disallow written for GPTBot does not apply to the other two.
Why does my * group not apply to ClaudeBot?
If your file has a group that names ClaudeBot, ClaudeBot follows that group and ignores *. Rules you wrote for every crawler under * then need repeating in the named group. The tester shows which group decides for each crawler.
Does robots.txt stop a crawler from reading a page?
No. robots.txt is a request that well-behaved crawlers follow; it locks nothing. A page that must stay private needs sign-in or another server control, and a search engine can still list a blocked address that other sites link to.
Want the fix made for you?
Answer Repair completes one agreed source-backed page or fix. Choose Live or Handoff before payment. Due 48 elapsed hours after Ready. Full refund on request within 30 days of delivery.
When you want us to do the work, every offer and its published price is on our offers page. More free checks: the ChatGPT Business Hub readiness check, the AI access checker, the crawler identity inspector and the sitemap generator, or see all free tools. Read next: How should you use robots.txt to control AI crawler access? and ChatGPT Business Hub explainer: crawler policy.