DataForAEOby Advance Labs

Use cases · Live

Track AI crawler visits to your site

See every ChatGPT, Claude and Perplexity bot that reads your pages, split into people asking, search indexing and training, with each hit checked by IP.

For SEO and content leads · Agencies running AEO · Developers · updated 2026-11-13

01 · The problem

Why this is hard today

AI assistants read your site in three very different ways: a person asks ChatGPT something and it fetches your page live, the assistant's search index refreshes, or a crawler collects pages for model training. Server logs and analytics lump these together or miss them entirely, most analytics tools never see bots at all, and any client can claim to be ChatGPT with one header. So most teams cannot answer a simple question: are people actually asking AI about our pages?

In figures

AI agents classified
15
ai-bots/classify.ts
of them fetch because a person asked
6
kind = user
vendors verified by IP range
3
ai-bots/verify.ts
IP addresses stored
0
schema-app-collector.sql

02 · How it works

How DataForAEO does it

  1. 01

    Install the collector

    Add a few lines to your Next.js proxy or a Cloudflare Worker. They send only requests whose User-Agent names a known AI bot, after the response, so pages never wait and ordinary visitors never leave your server.

  2. 02

    Every hit is classified

    Each request is matched to one of 15 AI agents and labelled as a user fetch, a search-index crawl or a training crawl.

  3. 03

    Every hit is checked by IP

    Requests claiming to be OpenAI, Anthropic or Perplexity bots are checked against the IP ranges those companies publish, and counted as verified, unverified or unknown.

  4. 04

    Read it per page

    The AI demand page in AEO Toolkit shows the last seven days: fetches for people, search-index crawls, training crawls, which pages and which bots.

What one stored hit looks like
{ "day": "2026-11-12", "path": "/pricing",
  "bot": "ChatGPT-User", "kind": "user",
  "verification": "verified", "hits": 1 }

03 · What you get

What you get

  • Which of your pages AI assistants read because a person asked
  • Which bots crawl you, and which of those hits are verified
  • Week-on-week change in AI demand per page
  • No IP addresses, query strings or human traffic stored
Get a collector keyWhat is data for AEO?

04 · Questions

Questions

Which AI crawlers does it track?

ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User, PerplexityBot, MistralAI-User, DuckAssistBot, Meta-ExternalFetcher, Meta-ExternalAgent, Amazonbot, Bytespider and CCBot. The full list with what each one does is on the AI crawler list page.

Does it slow my site down?

No. The snippet checks the User-Agent and, for AI bots only, sends the hit after the response has gone out, so the page never waits on it.

Can it see Google Gemini or Copilot?

Not separately. Google's and Microsoft's assistants fetch pages with their regular search crawlers, so their AI traffic cannot be told apart from search traffic by user agent.

What does it cost?

The collector and the AI demand page are free in AEO Toolkit. API access to the data is in beta; request access for details.