User fetch
A person asked the assistant something just now. This is real demand for the page.
ChatGPT-User · Claude-User · Perplexity-User · MistralAI-User · DuckAssistBot · Meta-ExternalFetcher
Registry · updated 2026-11-13
AI crawler list
Every AI bot user agent that reads your site, what it is doing, and whether it means a person asked. 15 agents from 9 companies, 6 of them fetching live for a user, 8 verifiable by published IP ranges.
Short answer: ChatGPT-User, Claude-User and Perplexity-User are people asking assistants about your page right now. OAI-SearchBot, Claude-SearchBot and PerplexityBot keep the assistants’ search indexes fresh. GPTBot, ClaudeBot and CCBot collect training data.
01 · The list
All 15 AI crawlers
| User agent | Company | Kind | What it does | IP list |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | User fetch | Fetches a page live when a ChatGPT user asks a question that needs it. | JSON |
| OAI-SearchBot | OpenAI | Search index | Crawls pages for ChatGPT search results. | JSON |
| GPTBot | OpenAI | Training crawl | Crawls pages that may be used to train OpenAI models. | JSON |
| Claude-User | Anthropic | User fetch | Fetches a page when a Claude user asks a question that needs it. | JSON |
| Claude-SearchBot | Anthropic | Search index | Crawls pages to improve Claude’s search results. | JSON |
| ClaudeBot | Anthropic | Training crawl | Crawls pages that may be used to train Anthropic models. | JSON |
| Perplexity-User | Perplexity | User fetch | Fetches a page when a Perplexity user’s question needs it. | JSON |
| PerplexityBot | Perplexity | Search index | Crawls pages to surface and link them in Perplexity answers. | JSON |
| MistralAI-User | Mistral AI | User fetch | Fetches a page when a Mistral Le Chat user’s question needs it. | none published |
| DuckAssistBot | DuckDuckGo | User fetch | Fetches pages to write DuckDuckGo’s AI-assisted answers. | none published |
| Meta-ExternalFetcher | Meta | User fetch | Fetches a specific page a Meta AI user asked about. | none published |
| Meta-ExternalAgent | Meta | Training crawl | Crawls pages for Meta AI training and indexing. | none published |
| Amazonbot | Amazon | Search index | Crawls pages for Amazon services such as Alexa answers. | none published |
| Bytespider | ByteDance | Training crawl | Crawls pages for ByteDance’s AI models. | none published |
| CCBot | Common Crawl | Training crawl | Crawls the open web for the Common Crawl corpus, used widely to train AI models. | none published |
02 · Three kinds
User fetch, search index, or training
Search index
The assistant’s search index refreshing. It decides whether you can be found, but it is not a person.
OAI-SearchBot · Claude-SearchBot · PerplexityBot · Amazonbot
Training crawl
Collecting pages for model training. It is not demand and does not send visitors.
GPTBot · ClaudeBot · Meta-ExternalAgent · Bytespider · CCBot
03 · Verify
How to tell a real AI crawler from a fake one
A User-Agent header is a claim any client can make. OpenAI, Anthropic and Perplexity publish the IP ranges their agents use, linked in the table above. To verify a request, check that its source IP falls inside the list for the agent it claims to be. A match is verified; a miss is a spoofed header or a stale list; an agent with no published list cannot be verified at all.
DataForAEO does this for every hit its collector sees, and keeps verified, unverified and unknown counts apart, so a scraper pretending to be ChatGPT never shows up as AI demand.
# Let assistants answer with your pages, keep training crawlers out.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /04 · Measure
See which AI crawlers read your pages
The DataForAEO collector counts these agents on your own site, per page per day, verified by IP, and shows the result in AEO Toolkit. It sends nothing but AI-bot requests and stores no IP addresses.
05 · Questions
AI crawler questions
What is the ChatGPT-User user agent?
ChatGPT-User is the user agent OpenAI uses when ChatGPT fetches a page live because a user asked a question that needs it. It is different from GPTBot (training) and OAI-SearchBot (search indexing). OpenAI publishes its IP ranges at openai.com/chatgpt-user.json.
Which AI crawlers mean a real person asked about my page?
The user-fetch agents: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot, Meta-ExternalFetcher. Each request is a person’s question landing on that page. Search-index and training crawlers are not people.
Should I block AI crawlers in robots.txt?
Blocking the user-fetch and search agents keeps assistants from reading or citing your pages, so most businesses allow them. Blocking training crawlers only affects whether your pages are used to train future models. The example on this page allows the first and blocks the second.
Can an AI crawler user agent be faked?
Yes. Any client can send any User-Agent. OpenAI, Anthropic and Perplexity publish IP range lists, so a request claiming to be one of their 8 agents can be checked against the real ranges. DataForAEO counts every hit as verified, unverified or unknown on that basis.
Is Googlebot or Bingbot on this list?
No. Google’s and Microsoft’s assistants fetch pages with their regular search crawlers, so their AI traffic cannot be told apart from search traffic by user agent. Those engines are measured through search data instead.