SAMPLE DATA, fictional company. This public example illustrates the report; it is not a client delivery.
Back to the Kalvenor Systems sample
Kalvenor Systems sample: bot access per engine (SAMPLE DATA, fictional company)
The section the findings document carries, here on a fictional site: whether ChatGPT, Claude, Perplexity, Google AI Overviews and Google AI Mode may read the site, and what to change where they cannot.
SAMPLE DATA
Bot access per engine
SAMPLE DATA. A fictional site, built to show each state. It is not a check of a real company.
3 pages checked, 1 excluded on purpose (sign-in). 3 of 5 engines have an access issue. Checked October 12, 2026 on https://kalvenor.example: robots.txt found (200); llms.txt not found, which no state on this page depends on.
| Engine | State | What to change |
|---|---|---|
| ChatGPT | Content not rendered | Serve the main text in the HTML, with server-side rendering or prerendering: /platform/dispatch carries 0 characters of visible text without JavaScript (an empty app root, a noscript message asking for JavaScript), and OpenAI's crawler documentation does not say its fetchers run JavaScript. |
| Claude | Blocked at the edge | Allow Claude-SearchBot and Claude-User in the bot rules of Cloudflare: Claude-SearchBot received a Cloudflare challenge page with HTTP 403 on the homepage. Where the provider publishes its IP addresses, allow by user agent and address together, so look-alike traffic stays blocked. |
| Perplexity | Blocked at robots.txt | Change robots.txt so PerplexityBot may read /platform/dispatch: "Disallow: /platform/" in the User-agent: PerplexityBot group blocks it. |
| Google AI Overviews | Reachable | Nothing to change: robots.txt lets Googlebot read the homepage and /platform/dispatch, and it received every page. |
| Google AI Mode | Reachable | Nothing to change: robots.txt lets Googlebot read the homepage and /platform/dispatch, and it received every page. |
- GPTBot is disallowed by robots.txt. OpenAI documents it as a training crawler, separate from OAI-SearchBot, so it does not change the state above.
- ClaudeBot received a Cloudflare challenge page with HTTP 403 on the homepage. Anthropic documents it as a training crawler, so it does not change the state above.
- Google-Extended is disallowed. Google says this token does not affect inclusion in Google Search, and that robots.txt rules for Googlebot are the control for AI features in Search.
- /platform/dispatch carries little text without JavaScript. Google says Googlebot renders JavaScript, so this matters less here than for the other engines; server-side rendering still helps every reader.
Pages checked: the homepage, /platform/dispatch, /signin (excluded on purpose: sign-in). The fetchers each provider documents: ChatGPT: OAI-SearchBot (search), ChatGPT-User (fetches a user asks for), GPTBot (training). Claude: Claude-SearchBot (search), Claude-User (fetches a user asks for), ClaudeBot (training). Perplexity: PerplexityBot (search), Perplexity-User (fetches a user asks for). Google AI Overviews and Google AI Mode: Googlebot (search), Google-Extended (robots.txt token only). Training crawlers and the Google-Extended token are reported, and never change a state.
Requests came from our servers with each provider's documented user agent. A firewall that verifies the engines' published addresses may answer the engines' own requests differently. Whether the main text is in the HTML is read from the HTML alone, without running JavaScript: visible text length beside app shell markers. It is a heuristic, not a browser render.