Free tool · unlimited
AI crawler checker & readiness scan
Check whether robots.txt lets GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 10 other AI crawlers read your site, plus rendering, schema, llms.txt, entity and agent signals.
The AI crawlers we check
Blocking a training crawler keeps you out of future model knowledge. Blocking a search or user crawler keeps you out of live AI answers today.
| User agent | Company | Used for |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Live browsing for users |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Claude search results |
| PerplexityBot | Perplexity | Perplexity answers |
| Google-Extended | Gemini and AI training | |
| Googlebot | Search and AI Overviews | |
| Bingbot | Microsoft | Bing and Copilot |
| Applebot-Extended | Apple | Apple Intelligence |
| meta-externalagent | Meta | Meta AI |
| Amazonbot | Amazon | Alexa and Rufus |
| CCBot | Common Crawl | Open dataset used by many LLMs |
| Bytespider | ByteDance | Doubao and TikTok AI |