AI crawlers and robots.txt: which bots to allow and which to block
By Hamza Rivera, updated
- Search bots decide whether your pages can be used in AI answers. Training bots are a separate choice, and each vendor says the settings work independently.
- OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training. Perplexity uses PerplexityBot. Anthropic uses Claude-SearchBot and ClaudeBot. Google uses Googlebot, with Google-Extended for some training controls.
- User-request bots such as ChatGPT-User and Perplexity-User may not follow robots.txt, since a person asked for the page.
- Robots.txt is one gate. A firewall or security plugin can block a bot before it reads the file.
- Pick a goal first: appear in AI answers, opt out of training, or both. Then edit one group at a time and check your logs.
The AI bots that matter for visibility are the search bots: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, Claude-SearchBot for Claude, and Googlebot for Google's AI features. Training bots such as GPTBot and ClaudeBot are a separate choice. You can allow search bots and still block training bots, because each rule works on its own. This guide lists each bot, what its owner says it does, and how to set robots.txt without hiding your site from AI answers.
For the wider picture of appearing in AI answers, see our ChatGPT SEO.
Why does the bot name matter?
Many sites paste a block list found online and disallow every bot with "AI" in the name. That can remove the site from the answers it hoped to appear in. OpenAI, Perplexity and Anthropic each run separate bots for search, for user requests and for model training, and each states in its own documentation that the settings are independent.
So the first job is to decide what you want. Do you want to appear in AI answers? Do you want your content kept out of model training? Those are two questions, and each has its own bot.
Which bot does what?
| Bot | Owner | What the owner says it does | Verdict |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search features | Allow to appear in ChatGPT search |
| GPTBot | OpenAI | Gathers content used to train generative AI foundation models | Your choice |
| ChatGPT-User | OpenAI | Visits pages for user actions; not used to decide Search appearance | Robots.txt may not apply |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ads | Only matters if you run ads |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results | Allow to appear |
| Perplexity-User | Perplexity | Fetches a page when a person asks | Generally ignores robots.txt |
| Claude-SearchBot | Anthropic | Improves search result quality for users | Allow to appear |
| Claude-User | Anthropic | Supports Claude users' requests | Blocking may reduce visibility |
| ClaudeBot | Anthropic | Collects web content that could contribute to training | Your choice |
| Googlebot | Crawls for Search, including AI features | Allow | |
| Google-Extended | Limits AI training and grounding in some other Google systems | Your choice |
The verdict column is a reading of what each owner says, not a ruling. Each company can change its bots and terms, so check the linked pages before you edit a live file.
What does OpenAI say about its bots?
OpenAI's bot documentation says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. It also says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. For search results, it says robots.txt changes can take about 24 hours to take effect.
GPTBot is for training. OpenAI says disallowing it signals that your content should not be used to train its generative AI foundation models. ChatGPT-User handles user-initiated actions, and OpenAI says it is not used to decide whether content may appear in Search and that robots.txt rules may not apply to it. For search opt-outs, the page points to OAI-SearchBot.
What does Perplexity say about its bots?
Perplexity says PerplexityBot is designed to surface and link websites in search results, and is not used to crawl content for AI foundation models. It recommends allowing it in robots.txt so your site appears in results. Perplexity-User visits pages to answer a person's question and generally ignores robots.txt, since a user asked for the fetch. Changes may take up to 24 hours to show. Our guide to getting cited by Perplexity covers the page side.
What does Anthropic say about its bots?
Anthropic lists three: ClaudeBot, Claude-User and Claude-SearchBot. ClaudeBot collects web content that could contribute to training. Claude-User supports Claude users, and disabling it may reduce your visibility for user-directed web search. Claude-SearchBot improves search result quality, and disabling it may reduce your visibility and accuracy in user search results. Anthropic also warns that blocking IP addresses may not reliably opt you out, so use robots.txt for the preference.
What does Google say about AI features?
Google's page on AI features names no separate bot for AI Overviews or AI Mode. It says there are no additional requirements to appear in them, and that a page must be indexed and eligible to be shown in Google Search with a snippet. Googlebot's robots.txt rules control crawling for Search. Google-Extended is the control for limiting AI training and grounding in some of Google's other systems.
That has a practical result. If you want to appear in Google's AI features, you need to stay indexable and snippet-eligible, and a rule meant for a training bot will not help or hurt that. Our guide on AI Overview citations for service businesses goes through the page-level side.
Want help with ChatGPT SEO?
Send us your site and we will record a short video of what we would fix first, or reply by email. Prefer to talk? A call is there too.
Get a free video audit See the serviceWhat should your robots.txt look like?
That depends on your goal, so here are three plain approaches.
- Appear everywhere and allow training: leave AI bots unrestricted. You need no special rules, only no accidental blocks.
- Appear in AI answers but opt out of training: allow OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot, and add a disallow rule for GPTBot, ClaudeBot and Google-Extended.
- Stay out of AI products: disallow the search bots and the training bots. Expect to lose AI mentions and some referral visits.
For the second approach, each bot gets its own group: a line naming the user agent, followed by a line that says Disallow with a slash. Put it at the top level of the domain, and repeat it for every subdomain, as Anthropic notes. After the edit, load the file in a browser and read it once more.
Think about who is affected by each choice. A local shop that lives on referrals from search usually gains from the second approach, since it keeps AI search visibility and still states a training preference. A publisher whose articles are the product may weigh the training question more heavily and may decide to restrict more. A business with no stated view can leave things open and revisit the file each quarter. Whatever you decide, write the reason next to the rule as a comment, so the next person to edit the file knows what it was meant to do.
Also remember that robots.txt is a request, not a lock. Vendors say their named bots honor it, but it does not hide a page from a person with the link. If something must stay private, put it behind a login instead.
What else can block a bot?
Robots.txt is only one gate. A firewall, a CDN setting, a security plugin or a login wall can turn a bot away before it reads the file. OpenAI, Perplexity and Anthropic each publish IP lists for their bots, and a site that filters by IP can use those lists to allow the right visitors. Anthropic cautions that blocking by IP may not persistently honor an opt-out, so the file remains the main tool.
Ask your host two questions: does the firewall treat unknown user agents as threats, and can bot protection be tuned by name? Our guide to robots.txt: what to block and what not to has more on the file.
Here is a made-up example of how a bad rule happens. A site owner reads an article that says to block AI crawlers, copies a list of twenty user agents into robots.txt and saves it. The list includes OAI-SearchBot and PerplexityBot next to the training bots. Six weeks later the owner asks why the business never shows up in ChatGPT search. The answer was in the file the whole time. Reading each name before you paste it takes two minutes and avoids the problem.
How do you confirm it worked?
Wait the 24 hours the vendors mention, then check your server logs for the bot names. See whether OAI-SearchBot and PerplexityBot appear and whether they get normal responses. A series of 403 or 429 codes means a firewall is in the way. Then ask a few customer questions in each AI product and watch for your pages in the sources. Our AI visibility tracking guide shows how to keep a log.
Keep a dated copy of each robots.txt version you publish, with a one-line note on why it changed. If your visibility drops, you can see what changed and when. Also recheck the file after a site migration, a new plugin or a host change, since those are common moments when a default file replaces your own.
Does llms.txt replace any of this?
No. An llms.txt file is a helpful list for language models, but it does not open a door that a firewall or robots.txt rule has closed. See what llms.txt is for its role.
Passieon is an independent agency and is not affiliated with OpenAI, Perplexity, Anthropic or Google. Check the vendor pages below before changing a live site.
What should you do this week?
Open your robots.txt, write down which of the bots in the table it mentions, and decide whether you want AI search visibility, training opt-out, or both. Edit one group at a time, retest the file, and read your logs a day later.
Sources
- OpenAI: Overview of OpenAI crawlers, read November 17, 2026.
- Perplexity: Perplexity crawlers, read November 17, 2026.
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?, read November 17, 2026.
- Google Search Central: AI features and your website, read November 17, 2026.
Frequently asked questions
Should I block GPTBot?
GPTBot is OpenAI's training crawler. Blocking it signals that your content should not be used for training, and it is separate from OAI-SearchBot, which handles ChatGPT search.
Will blocking OAI-SearchBot remove me from ChatGPT?
OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
Do ChatGPT-User and Perplexity-User follow robots.txt?
OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person requested the visit.
Is there a special bot for Google AI Overviews?
Google's page names none. Googlebot's rules control Search crawling, and a page must be indexed and snippet-eligible to appear. Google-Extended limits AI training and grounding in some other systems.
How long do robots.txt changes take?
OpenAI and Perplexity each say about 24 hours for search systems to adjust.
Can a firewall block AI bots even if robots.txt allows them?
Yes. Security plugins, CDNs and firewalls can turn bots away before they read the file, so check both.
One last thing
Before you paste any block list from the web, read the bot names in it. A rule meant to stop training can also hide you from AI search.
Want this done for your site? See what our ChatGPT SEO page covers, or read the client results.
Related guides
- White label SEO agreement: the clauses that protect your agency and your clients The clauses to check in a white label SEO agreement: scope, confidentiality, no client contact, ownership, Google policy compliance and exit terms.
- White label link building: what it is and how to check what you are buying What white label link building is, what Google's spam policies say about links, and how to check a placement before a client's site is attached to it.
- How to get cited by Perplexity: crawler access, answer-first pages and checks How to get cited by Perplexity: allow PerplexityBot, put the answer first, show your method, and check which questions cite you each month.