Who crawls your site for AI, and what blocking them does
AI companies send bots to read websites for three different reasons. Blocking one kind can keep your pages out of model training; blocking another can take you out of AI answers. This page lists every bot we track, what it is for, and how to tell it apart.
25 bots · 12 companies
Not every AI bot does the same job
Each company runs separate bots, with separate names, for separate jobs. That is what lets you allow one and block another.
Training crawlers
Collect public pages to train future AI models. They crawl in bulk and do not send visitors back.
Your pages stay out of future training sets. Answers that cite live pages are not affected, because those come from the search bots.
Search crawlers
Build the index an AI assistant searches when it answers a question, the same way a search engine does. This is where citations come from.
The assistant can no longer find or cite your pages in its answers. For most businesses this is the one to keep open.
On-request fetches
Open one page because a person asked their assistant to, for example by pasting your link into a chat. One request at a time, not a crawl.
The assistant tells the person it could not open your page, at the moment they were looking at you.
Training crawlers
Collect public pages to train future AI models. They crawl in bulk and do not send visitors back.
| Bot | Appears in traffic as | robots.txt token | Published IP list |
|---|---|---|---|
| GPTBot | GPTBot | GPTBot | |
| ClaudeBot | ClaudeBot | ClaudeBot | |
| anthropic-ai | anthropic-ai | ||
| Google-Extended | Google-Extended | ||
| Applebot-Extended | Applebot-Extended | ||
| CCBot | CCBot | CCBot | |
| cohere-ai | cohere-ai | cohere-ai | |
| Meta-ExternalAgent | meta-externalagent | Meta-ExternalAgent | |
| Bytespider | Bytespider | Bytespider |
Search crawlers
Build the index an AI assistant searches when it answers a question, the same way a search engine does. This is where citations come from.
| Bot | Appears in traffic as | robots.txt token | Published IP list |
|---|---|---|---|
| OAI-SearchBot | OAI-SearchBot | OAI-SearchBot | |
| OAI-AdsBot | OAI-AdsBot | OAI-AdsBot | |
| Claude-SearchBot | Claude-SearchBot | Claude-SearchBot | |
| PerplexityBot | PerplexityBot | PerplexityBot | |
| Googlebot | Googlebot | Googlebot | |
| Applebot | Applebot | Applebot | |
| Amazonbot | Amazonbot | Amazonbot | |
| Amzn-SearchBot | Amzn-SearchBot | Amzn-SearchBot | |
| DuckAssistBot | DuckAssistBot | DuckAssistBot | |
| Bingbot | bingbot | Bingbot |
On-request fetches
Open one page because a person asked their assistant to, for example by pasting your link into a chat. One request at a time, not a crawl.
| Bot | Appears in traffic as | robots.txt token | Published IP list |
|---|---|---|---|
| ChatGPT-User | ChatGPT-User | ChatGPT-User | |
| Claude-User | Claude-User | Claude-User | |
| Claude-Web | Claude-Web | Claude-Web | |
| Perplexity-User | Perplexity-User | Perplexity-User | |
| Amzn-User | Amzn-User | Amzn-User | |
| Meta-ExternalFetcher | meta-externalfetcher | Meta-ExternalFetcher |
Block training, stay in AI answers
This is the setup most businesses want: training crawlers kept out, search crawlers and on-request fetches let in. It is built from the tables above, one group per kind.
# Training crawlers: keep out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: cohere-ai User-agent: Meta-ExternalAgent User-agent: Bytespider Disallow: / # AI search crawlers: stay findable and citable User-agent: OAI-SearchBot User-agent: OAI-AdsBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Applebot User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: DuckAssistBot User-agent: Bingbot Allow: / # Fetches a person asked their assistant for User-agent: ChatGPT-User User-agent: Claude-User User-agent: Claude-Web User-agent: Perplexity-User User-agent: Amzn-User User-agent: Meta-ExternalFetcher Allow: /
- robots.txt is a request, not a lock. The companies listed here say their crawlers follow it; anything else needs a firewall rule.
- Some companies say robots.txt may not apply to a page a person asked their assistant to open, since that works like a click.
- A firewall or bot protection setting can block a bot that robots.txt allows. If the bot gets a challenge page, it sees nothing.
- Keep your existing rules for other crawlers. Add these groups next to them; a bot follows the group that names it most specifically.
Check what AI bots actually get from your site
Reading robots.txt is half of it. The free audit reads your robots.txt rules for each AI bot, then requests your page as a browser and as four AI bots to see whether your firewall turns them away.
Free AI-readiness audit
Your robots.txt rules per bot, the firewall check, llms.txt, structured data and more. No account needed.
What the firewall check sends
Seen +https://knownwise.com/ai-bots/probe in your logs? That was us. Here is what we request and how to block it.
Common questions
Does blocking GPTBot remove my site from ChatGPT answers?
No. GPTBot collects training data. ChatGPT finds pages for its answers with OAI-SearchBot, and opens a page a person asks for as ChatGPT-User. Blocking GPTBot keeps your pages out of future training without taking you out of ChatGPT search.
Does blocking Google-Extended affect Google Search?
No. Google-Extended is a robots.txt token only: Google never sends it as a user agent, and it does not change how Googlebot crawls or ranks your site. It controls whether Google may use your content to train and ground its Gemini models.
Do AI bots follow robots.txt?
The companies listed here say their crawlers do. Fetches a person asks for can be treated differently, and robots.txt never stops a bot that chooses to ignore it. To enforce a block, use a firewall rule.
How can I tell a real AI bot from a fake one?
Anyone can copy a user agent. Check the request's IP address against the company's published IP list, linked in the tables above. A request that claims to be GPTBot from an address outside OpenAI's list is not GPTBot.
Why would my firewall block AI bots I allow in robots.txt?
Many CDNs and bot protection services challenge or block AI crawlers by default or with a single setting. The bot gets a challenge page instead of your content, whatever robots.txt says. The free audit tests this for you.
How do I see which AI bots visit my site?
Your server or CDN logs record the user agent of every request; search them for the tokens in the tables above. Knownwise paid plans count AI bot visits per day for you, by bot and by page.
Find out which AI bots can read your site
The free audit checks your robots.txt rules and your firewall for each major AI bot, in about a minute.