AI Crawlers: The Complete List of Documented Bots

If you read your server logs, the AI traffic is not hypothetical: named bots from OpenAI, Anthropic, Perplexity, Google, and Microsoft fetch pages daily, each under a documented user-agent token with an official policy page behind it. This list covers every officially documented token — 12 across 5 companies — what each one is for, how it treats robots.txt, and how to tell a real one from a spoof. It is the reference we use when reviewing our own fleet's logs.

Which AI crawlers are documented right now?

Twelve tokens, five companies — the table below is the complete officially documented set as of August 2026. Anything else claiming to be an AI bot in your logs is either undocumented or fake.

Documentation status as of August 2026; each vendor's page is the live authority, and tokens do get added — OpenAI's OAI-AdsBot is a 2025-era addition most lists still miss.

What are the three jobs an AI bot can have?

Every documented AI token does exactly one of three jobs — training, search indexing, or user-requested fetching — and the job is what decides whether blocking it costs you visibility.

Training crawlers (GPTBot, ClaudeBot, Google-Extended as a control) collect content for building models. OpenAI's docs say disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models"; Anthropic describes ClaudeBot as "collecting web content that could potentially contribute to their training." Blocking these affects future model corpora, not whether an answer engine can show your pages today.

Search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot) build the indexes answer engines retrieve from. These are the ones with direct visibility stakes: OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." The training-vs-search distinction is unpacked decision by decision in GPTBot vs OAI-SearchBot.

User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) retrieve a page because a human asked the assistant to. They are the robots.txt gray zone: OpenAI says rules "may not apply" to ChatGPT-User, and Perplexity is blunter — Perplexity-User "generally ignores robots.txt rules" since a user requested the fetch. Anthropic is the outlier that documents Claude-User as respecting robots.txt.

Which of these bots actually show up in server logs?

The training crawlers dominate, and the gap between fetching and referring is enormous. Cloudflare's crawl-to-click analysis (January–July 2025) measured how many pages each platform crawls per referral it sends back: roughly 38,066:1 for Anthropic, 1,091:1 for OpenAI, and 195:1 for Perplexity in July 2025, against about 5:1 for Google. The metric has its own page — the crawl-to-refer ratio — because it is becoming the standard way to price crawler access.

Our own logs match the direction of that data: across our three production builds, ClaudeBot, GPTBot, and PerplexityBot all appear as regular visitors [our data] — and Cloudflare's ratios above are the published measure of how lopsidedly training crawls outnumber referring surfaces. We have not published per-bot request counts yet; when we do, they will be dated log extracts, not estimates.

One log-reading caution from running this in production: a bot's fetch volume tells you nothing about citation likelihood. Heavy ClaudeBot traffic is a training-corpus signal, not a Claude-answers signal — Cloudflare's thousands-to-one crawl-to-refer ratios are exactly why the two must not be conflated.

How do you verify a bot is real and not a spoof?

Check the source IP against the vendor's published list — the user-agent string proves nothing, because any scraper can send GPTBot as its user agent. Verification paths, per the official docs:

• Anthropic publishes its crawler IPs in a JSON file (referenced from its crawler support article) for source verification.
• Perplexity publishes separate IP JSON files for PerplexityBot and Perplexity-User, linked from its crawler docs.
• OpenAI publishes IP ranges for its bots in the same documentation that defines them.
• Google documents three checks: the user-agent header, the source IP, and the reverse-DNS hostname of that IP.

In practice, filtering log lines to vendor-published IPs is the difference between measuring AI crawler behavior and measuring whoever felt like impersonating it that week. Spoofed AI user agents are common enough that any unverified analysis should be treated as contaminated.

Should you block any of them?

Sometimes — but it is a per-token business decision, not a hygiene default, and the honest answer depends on how you make money. A publisher monetizing pageviews faces that 38,066:1 extraction ratio directly and can reasonably block training crawlers. A lead-generation or authority site — our situation — mostly benefits from being retrievable, which is why we currently allow the documented AI crawlers across our builds [our data].

Blanket-blocking "AI bots" as a category is the one clearly wrong move, because the category mixes three different jobs: block OAI-SearchBot along with GPTBot and you have removed yourself from ChatGPT search answers to stop a training crawler. The full per-bot, per-business-model decision table lives in the robots.txt decision framework, and the strategic version of the question — what citations are actually worth trading crawl access for — runs through everything in our generative engine optimization guide.