Free tools / AI access checker
AI access checker: can AI crawlers read this page?
Our AI access checker tests one page the way AI crawlers meet it. It reports robots.txt rules for 11 AI and search crawlers, the X-Robots-Tag header and robots meta tags, the HTTP status a browser gets beside the status a user agent naming an AI crawler gets, and whether the main text is in the raw HTML.
We read robots.txt before anything else and follow it, then read your page within eight seconds: up to three redirects, each checked again and each to a public HTTPS address, and at most 1.5 MB. HTTPS websites are supported; other addresses are refused. We request the page twice: once with a browser user agent and once with a user agent that names the crawler you choose and Broadcastwell.
No email is needed; runs are limited per network and per day, and by the site's daily allowance.
We do not save your website, company name, page content or result. We keep daily totals and hashed network counters to enforce limits. A shared result stays in its link, not on our server.
What we check and why
robots.txt rules for 11 crawlers
At the exact path you enter, we show whether robots.txt allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended and Bingbot, and which group and rule decide it.
X-Robots-Tag and robots meta tags
noindex, nosnippet, max-snippet:0 and similar rules sit in the page or its headers, not in robots.txt. They can keep a page out of search results or limit how much text is shown, even when robots.txt allows the crawler.
A browser user agent and an AI user agent
Some firewalls and bot filters answer requests that name an AI crawler with 403 or a challenge page. Comparing the two answers shows whether your server treats them differently. Our request also names Broadcastwell, and a filter that checks IP addresses sees it is not the real crawler, so treat a difference as a lead to check in your firewall settings.
Main text in the raw HTML
Crawlers that do not run scripts read the HTML your server sends. We count the words of main text in that HTML; under 100 words usually means scripts build the page.
Questions
Why does my page return 403 to an AI crawler but 200 in my browser?
A firewall, a CDN bot setting or a security plugin is usually answering requests by user agent. Look for a rule that blocks AI crawlers or unverified bots. If you want those crawlers to read the page, allow them there as well as in robots.txt.
What is the difference between robots.txt and a noindex tag?
robots.txt asks a crawler not to fetch a path. A noindex tag or X-Robots-Tag header lets the crawler fetch the page but asks it not to list the page. A crawler that robots.txt keeps out never sees a noindex on that page.
Does passing this check mean ChatGPT or Claude will cite my page?
No. It shows that nothing on your site stops the crawlers from reading the page. Whether an engine names or cites you takes a dated measurement of real buyer questions, with a receipt for each answer.
Want the fix made for you?
Answer Repair completes one agreed source-backed page or fix. Choose Live or Handoff before payment. Due 48 elapsed hours after Ready. Full refund on request within 30 days of delivery.
When you want us to do the work, every offer and its published price is on our offers page. More free checks: the ChatGPT Business Hub readiness check, the robots.txt tester, the crawler vs browser view, the crawler identity inspector and the agent readiness grader, or see all free tools. Read next: What do AI crawlers do, and how can site owners control access?, Bot access and AI search visibility and ChatGPT Business Hub explainer: crawler policy.