Learn
How to check AI crawler access to your website
To check whether AI crawlers can reach your site, read robots.txt for each crawler by name, request your key pages and confirm they return full HTML, then check firewall and bot-management rules and your server or CDN logs for real crawler requests. Search access and model-training access are separate choices.
Key points
- Check the search crawlers behind AI answers first: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot.
- A robots.txt rule is one layer; a firewall or challenge page can still block a crawler.
- Changing a user-agent string in a test does not prove a real crawler gets through.
- Blocking a training crawler is a different decision from blocking a search crawler.
In more detail
Checking AI crawler access takes four short steps. First, find out which crawlers matter. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so answer engines can cite them. Second, read your robots.txt for each of those names, because a rule for one bot does not cover another. Third, request your key pages and confirm that each one returns full HTML, not a challenge page. Fourth, look at firewall rules, bot management and server logs for real crawler requests. A robots.txt rule is only one layer, and a firewall can still block a bot that robots.txt allows. A test with a changed user-agent string proves little about the real crawler. Training crawlers such as GPTBot and ClaudeBot are a separate choice, and blocking them is not the same as blocking search. Repeat the check after any change to your firewall or CDN.
Which AI crawlers should you check?
Check the crawlers that index pages for AI search first, then decide separately about user-triggered fetchers and training crawlers. Each vendor names its bots and what they are for in its own documentation. The table lists the ones those vendors document today; always confirm against the linked pages, because vendors change them.
| Vendor | Search | User-triggered | Training or control |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | None: PerplexityBot is not used for training |
| Googlebot | See Google’s list | Google-Extended (a control token, not a separate crawler) |
User-triggered fetchers visit a page because a person asked an assistant about it. OpenAI and Perplexity say robots.txt rules may not apply to theirs, so a robots.txt block does not reliably stop them.
How do you read robots.txt for a specific crawler?
Open yoursite.com/robots.txt and find the group that names the crawler. If none does, the crawler follows the group for all user agents, written as an asterisk. Within the group, the most specific matching rule wins, so a longer Disallow path can override a broad Allow. RFC 9309 defines these rules.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /That example is only an illustration of the syntax: it welcomes OpenAI’s search crawler and opts out of its training crawler. Which choice is right for your site is your decision.
How do you test that a page loads for a crawler?
Request the page the way a crawler would and read what comes back, not just the status code. A 200 response that contains a challenge page or an empty shell is still a failure. The steps below work for one page and can be repeated whenever the hosting setup changes.
- Read robots.txt. Find the group for each crawler in the table and note any Disallow rule that matches your key pages.
- Request the page. Fetch it with the exact user-agent string from the vendor’s documentation, for example with curl and its -A option, and keep the full response.
- Read the response. Confirm a 200 status and that the HTML contains the page’s real headings and text, not a challenge, a login wall or an empty application shell.
- Check the firewall and bot rules. Look at your CDN, firewall and bot-management settings for rules that challenge or deny automated traffic, and allow the vendors’ published IP ranges where they provide them.
- Find real requests in your logs. Search server or CDN logs for each crawler name and confirm the source IPs match the ranges the vendor publishes.
- Allow time, then recheck. OpenAI says search-related robots.txt changes can take about 24 hours to take effect, so repeat the test after a day.
curl -s -o /dev/null -w "%{http_code}\n" -A "<user-agent string from the vendor docs>" https://example.com/your-pageWhy might a crawler be allowed in robots.txt but still blocked?
Because robots.txt is only a request, and other layers decide what the server actually returns. A CDN rule, a web application firewall, rate limiting or a bot challenge can deny a legitimate crawler, and a page that needs JavaScript to show its text can look empty. Spoofing a user-agent string tests none of the IP-based rules.
Vendors suggest pairing the user-agent with their published IP ranges. Perplexity documents firewall guidance, and Google explains how to verify requests that claim to be Google. Do not disable site security broadly to let a user-agent string in.
Does blocking GPTBot or ClaudeBot remove you from AI search?
According to the vendors, no, because they separate the bots. OpenAI describes GPTBot as the training crawler and OAI-SearchBot as the one for ChatGPT search. Anthropic likewise separates ClaudeBot (training) from Claude-SearchBot (search). Blocking the search bots, by contrast, can reduce how often a site appears in those answers.
Check each vendor’s page before you change a rule: OpenAI, Anthropic, Perplexity and Google. Google-Extended, for example, does not affect inclusion in Google Search.
Where does Xaeo fit?
Xaeo does not check your site today: it is a pre-launch product, and the public demo is illustrative and does not scan the domain you enter. Its planned fix reports start from observed AI answers for each client. Use the steps above for a manual check. Beta opens November 2026.
For the wider picture, read which crawlers matter for AI search and the GEO explainer.