GPTBot, PerplexityBot, ClaudeBot: who to allow in robots.txt

Updated 2026-08-03

Why robots.txt is the first checkpoint for AI bots

Over two weeks in July, a three-store panel in our AI-traffic index logged around 1,005,146 AI-bot hits — the top vendor alone accounted for 606,017, roughly 34,000 a day. This is no longer background noise: language models actively read catalogs to answer shoppers. Your robots.txt, at the root of the site, is the first place you decide which of these bots to let in.

One caveat up front: robots.txt is an agreement, not a lock. Polite bots read it and obey, but the file itself blocks nothing physically. More on that at the end.

The three roles of AI bots

Not all AI bots are the same. Each vendor runs several, with different jobs:

Training crawlers collect content to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), Applebot-Extended (Apple), CCBot (Common Crawl), Bytespider (ByteDance).
Answer bots show up at query time to assemble a fresh answer for the user right now: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot.
On-behalf-of-a-human agents fire when a real shopper asks the assistant to open your link: ChatGPT-User, Perplexity-User, Claude-User.

For a store, the second and third groups matter most — they are what puts you into the answer a buyer actually sees. Training is a matter of taste: some allow it for brand presence, some block it to avoid giving content away.

User-Agent reference

Vendor	User-Agent	What it does
OpenAI	`GPTBot`	model training
OpenAI	`OAI-SearchBot`	answers in ChatGPT search
OpenAI	`ChatGPT-User`	a human opened the link via ChatGPT
Anthropic	`ClaudeBot`, `anthropic-ai`	model training
Anthropic	`Claude-User`	a human opened the link via Claude
Perplexity	`PerplexityBot`	index for answers
Perplexity	`Perplexity-User`	a human opened the link via Perplexity
Google	`Google-Extended`	Gemini / AI training
Apple	`Applebot-Extended`	Apple AI training
ByteDance	`Bytespider`	data collection
Common Crawl	`CCBot`	open training dataset

Training vs answers: the Google-Extended catch

The most common mistake is assuming that "block AI" and "stay in search" are the same thing. They are not.

Google-Extended is not a separate crawler. It is a switch token: it tells Google whether your content may be used to train Gemini. Regular Googlebot keeps indexing you, and your search rankings are unaffected. You can opt out of Google's AI training without losing a drop of SEO.

Same logic with OpenAI. Block GPTBot and your content stops feeding training. But if you also close OAI-SearchBot, you drop out of ChatGPT search answers. If you want "don't train me, but keep me in answers," block GPTBot and leave OAI-SearchBot open.

robots.txt examples

The open strategy — for a store that wants to sell through AI. Let everyone in:

# Let all AI bots in
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Classic search engines — always open
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Sitemap: https://your-store.com/sitemap.xml

The "don't train, but stay in answers" strategy:

# no content for training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# but stay in live ChatGPT and Perplexity answers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# explicitly evict the greedy scraper
User-agent: Bytespider
Disallow: /

Never touch Googlebot or Bingbot

The temptation to write User-agent: * and Disallow: / to "shut out all AI" ends in disaster: * catches the classic search engines Googlebot and Bingbot, and your site falls out of ordinary search. Google traffic still drives the bulk of your orders — killing it to fight AI is self-harm.

The rule is simple: Googlebot and Bingbot stay open, always. The Botmetria firewall has this fuse built in — they cannot be blocked, even by accident.

Why robots.txt alone is not enough

robots.txt runs on the honor system, which leaves two holes:

Anyone can forge the User-Agent string. A scraper sends the header GPTBot and your Allow for GPTBot waves it through. And Bytespider plus some aggressive bots have historically ignored robots.txt entirely.
A block in the file is not a real block. A polite bot reads Disallow and leaves. An impolite one does not.

That is why Botmetria does not take the User-Agent at its word. Every AI bot is checked against the vendor's published IP ranges — a real GPTBot comes from OpenAI's IPs, and a fake is caught and cut by the firewall. Bots that execute JavaScript are treated as humans rather than crawlers, so genuine visits do not get mixed up with data collectors.

Start small — see who is already walking your domain. A free Botmetria audit checks your robots.txt and shows which bots you are open to; then compare yourself against the market in the MD/RO store rating.

FAQ

If I block GPTBot, will I disappear from Google search?

No. GPTBot is OpenAI's crawler and has nothing to do with Google. Even blocking Google-Extended leaves your rankings intact — regular Googlebot keeps indexing you. SEO only suffers if you accidentally close Googlebot or Bingbot, which you must never do.

If I block every AI bot, can I still appear in ChatGPT answers?

No. To show up in ChatGPT search answers, keep OAI-SearchBot open, and for Perplexity keep PerplexityBot open. Block only the training crawlers (GPTBot, Google-Extended) while leaving the answer bots allowed.

Does robots.txt really block AI bots?

Only the polite ones. The file is an agreement: the User-Agent string is trivial to forge, and some bots (Bytespider, for one) ignore robots.txt. Reliable blocking happens at the firewall, verifying the vendor's IP ranges — which is what Botmetria does.

Related guides

How to check whether GPTBot visits your site

Check your store for free

A 60-second audit shows how ready your store is for AI agents — and exactly what to fix first.

Run the free audit