Is GPTBot blocked on your site? How to check robots.txt for AI crawlers

Which AI crawlers exist, what blocking each one does, and a two-minute check of your robots.txt so ChatGPT, Claude, Gemini and Perplexity can find you.

By Farzad Golestan, Founder of AnswerSeat 8 min read
/robots.txtUser-agent: GPTBotDisallow: /BLOCKEDUser-agent: OAI-SearchBotAllow: /SEENUser-agent: Claude-SearchBotAllow: /SEENUser-agent: PerplexityBotAllow: /SEENANSWERSEAT.COM / BLOG

Short answer

Open yourdomain.com/robots.txt and look for groups that name AI crawlers, such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot or Google-Extended, followed by "Disallow: /". Also check whether a "User-agent: *" group disallows everything. To appear in AI search answers, the search crawlers matter most: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking training crawlers such as GPTBot is a separate business choice.

What does robots.txt control, and what does it not?

robots.txt is a public text file at the root of your domain. It tells crawlers which paths they may fetch. OpenAI, Anthropic, Google and Perplexity all say their crawlers read it and follow it.

Two limits matter. First, robots.txt is a request, not a lock: a firewall or bot protection at your host can still turn a crawler away even when robots.txt allows it. Second, fetchers that open a page because a user asked for it, such as ChatGPT-User, are treated differently. OpenAI says robots.txt rules may not apply to them, because a person started the request.

Which AI crawlers should you know?

Each company separates crawling for training from crawling for search. That lets you allow one and block the other.

CrawlerCompanyWhat it doesIf you block it
GPTBotOpenAICollects content that may be used to train modelsYour content is left out of future training. ChatGPT search is controlled separately.
OAI-SearchBotOpenAIFinds pages for ChatGPT search answersYour pages are left out of ChatGPT search answers.
ChatGPT-UserOpenAIOpens a page when a user asks ChatGPT toNot automatic crawling; robots.txt may not apply.
ClaudeBotAnthropicCollects content that may be used for trainingYour future content is left out of training.
Claude-SearchBotAnthropicIndexes content for Claude search answersMay reduce your visibility in Claude search results.
Claude-UserAnthropicFetches a page when a Claude user asks a questionMay reduce visibility in answers Claude looks up for users.
PerplexityBotPerplexityBuilds the index Perplexity answers fromYour site may not appear in Perplexity answers.
Google-ExtendedGoogleNot a crawler: a robots.txt token for Gemini training and groundingDoes not change your Google Search ranking. AI Overviews use regular Googlebot.

How do you check your robots.txt in two minutes?

  1. Open https://yourdomain.com/robots.txt in your browser. If the page does not exist, there are no rules, and every crawler may fetch everything.
  2. Find each block that starts with "User-agent:". The lines below it, up to the next "User-agent:", are the rules for that crawler.
  3. Look for blocks that name an AI crawler from the table. "Disallow: /" under one of them blocks that crawler from your whole site.
  4. Look at the "User-agent: *" block. "Disallow: /" there blocks every crawler that does not have its own block, including AI crawlers you never named.
  5. Remember the matching rule: a crawler follows the block that names it and ignores the "*" block. A crawler with no block of its own follows "*".
Example: keep content out of OpenAI training, but stay visible in ChatGPT, Claude and Perplexity search.
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Should you block AI training crawlers?

It depends on what you sell. Publishers whose content is the product often block training crawlers such as GPTBot, ClaudeBot and Google-Extended. Most businesses that want to be recommended have the opposite goal: they want assistants to know who they are.

Keep the two decisions apart. Blocking training affects what future models know from memory. Blocking search crawlers removes you from the answers that use live search today, and those are the answers buyers see for most product questions. If you only change one thing, make sure OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed.

Why can AI crawlers be blocked even when robots.txt allows them?

  • Bot protection at your host or CDN that challenges or blocks automated visitors, including AI crawlers.
  • Firewall rules that block by IP range. Anthropic advises against blocking its crawler by IP, because that can stop it from reading your robots.txt at all.
  • Content behind a login, a cookie wall or a country block.
  • Pages that only show their text after JavaScript runs. Many AI crawlers read the raw HTML.

If robots.txt looks right but you never appear in AI search answers, ask your host whether a bot protection setting is on, and check your server logs for the crawler names in the table.

How long until a change takes effect?

Crawlers re-read robots.txt on their own schedule. OpenAI says its search systems need about 24 hours to adjust after you change robots.txt. Allowing a crawler does not make it visit at once, and new pages still need to be found through links or a sitemap.

To test all of this in one go, our free AI crawler checker reads your robots.txt for the main AI crawlers and checks whether your content is visible in the HTML. If you are allowed everywhere but still rarely named, the cause is usually elsewhere: see why ChatGPT recommends your competitor instead of you.

Questions people also ask

Does blocking GPTBot remove my site from ChatGPT?

No. OpenAI controls training and search with separate crawlers. Blocking GPTBot keeps your content out of future training, while ChatGPT search uses OAI-SearchBot. You can block one and allow the other.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls the use of your content for Gemini training and grounding. Google says it does not affect Google Search, and AI Overviews are built from pages that regular Googlebot crawls.

Can robots.txt keep my content private?

No. robots.txt is a public request that well-behaved crawlers follow. Anyone can read it, and it does not stop a person or a badly behaved bot. Use a login for content that must stay private.

Do I also need an llms.txt file?

llms.txt is a separate, optional file that summarizes your site for language models. Support varies between AI tools and Google says it does not need it. robots.txt access comes first.

See which brands ChatGPT, Gemini and Claude name instead of you.

The free audit runs buyer questions for you and hands you a prioritised list of fixes.

Run my free audit

No credit card. Results in 3 to 5 minutes.

Sources

Comments

No comments yet. Questions and experiences are welcome.

Sign in to comment

Comments are open to AnswerSeat members. We email you a one-click sign-in link; new here, the same link creates your free account.

By continuing you agree to the Terms of Use and Privacy Policy.

Keep reading