AI Crawler Checker: can answer engines reach you?
Test your live robots.txt against 19 AI and search crawlers. If an answer engine's bot is blocked, it cannot cite you.
AI crawlers and what they do
| robots.txt token | Operator | Type | Used for |
|---|---|---|---|
| Googlebot | Search engine | Crawls for Google Search, which includes AI Overviews and AI Mode. Blocking it removes your pages from Google. | |
| Google-Extended | Control token | Not a separate crawler. A robots.txt token that decides whether content Googlebot fetches may be used to train and ground Gemini models. It does not affect Google Search or AI Overviews. | |
| Bingbot | Microsoft | Search engine | Crawls for Bing Search, whose index also grounds Microsoft Copilot answers. |
| OAI-SearchBot | OpenAI | AI search index | Indexes pages so they can be shown and linked in ChatGPT search results. |
| ChatGPT-User | OpenAI | User request | Fetches a page when a ChatGPT user or a GPT action asks for it. It is not used for automatic crawling. |
| GPTBot | OpenAI | Model training | Crawls content that may be used to train OpenAI foundation models. Blocking it does not remove you from ChatGPT search. |
| Claude-SearchBot | Anthropic | AI search index | Indexes content to improve the quality of search results in Claude. |
| Claude-User | Anthropic | User request | Fetches pages when a Claude user asks a question that needs them. |
| ClaudeBot | Anthropic | Model training | Crawls content that may be used to train Anthropic models. |
| PerplexityBot | Perplexity | AI search index | Indexes websites so they can be surfaced and linked in Perplexity answers. Perplexity states it is not used to train foundation models. |
| Perplexity-User | Perplexity | User request | Fetches a page when a Perplexity user asks a question that needs it. |
| Applebot | Apple | Search engine | Crawls for Apple search features in Siri, Spotlight and Safari. |
| Applebot-Extended | Apple | Control token | Not a separate crawler. A token that decides whether content Applebot fetches may be used to train Apple generative models. It does not stop Applebot crawling. |
| DuckAssistBot | DuckDuckGo | AI search index | Fetches pages to answer questions in DuckDuckGo AI-assisted answers. |
| Amazonbot | Amazon | AI search index | Crawls to improve Amazon products and services, including answers given by Alexa. |
| MistralAI-User | Mistral AI | User request | Fetches pages when a Le Chat user asks for them. |
| Meta-ExternalAgent | Meta | Model training | Crawls content for AI training and AI features in Meta products. |
| CCBot | Common Crawl | Model training | Builds the open Common Crawl web archive, which many language models are trained on. |
| Bytespider | ByteDance | Model training | Crawls content for ByteDance AI products and model training. |
Allow answer engines, block training
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /A bot with its own User-agent group ignores the * group, and the longest matching rule wins. Once crawlers can reach you, check what they find with the AEO score checker.
AI crawler FAQ
How do I check if ChatGPT can crawl my site?
Enter your domain above. The checker reads your live robots.txt and reports whether OAI-SearchBot (ChatGPT search), ChatGPT-User and GPTBot (training) are allowed. OAI-SearchBot must be allowed for ChatGPT search to cite you.
Does blocking GPTBot hurt my AEO?
No. GPTBot is a training crawler. ChatGPT search uses OAI-SearchBot. You can block training while allowing search.
Which bots matter for answer engines?
Googlebot (AI Overviews), Bingbot (Copilot), OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User (Perplexity), and Claude-SearchBot and Claude-User (Claude).
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls use of content for Gemini training and grounding. AI Overviews are part of Google Search and use Googlebot.
More free AEO tools
AEO Score Checker
Score any page on 22 answer-engine factors, compared with the competitors AI actually cites.
Open tool →AI Citation Gap Finder
See who AI cites for your question, whether you are cited, and what the winners do differently.
Open tool →FAQ Schema Generator
Turn your questions and answers into valid FAQPage JSON-LD in seconds.
Open tool →