ClaudeBot, Claude-User, Claude-SearchBot: What Anthropic's Crawlers Do

ClaudeBot shows up in server logs with a frequency that makes operators nervous, and the first search result explaining it is usually a blog post selling a blocking tool. The primary source is calmer: Anthropic documents three web agents, what each does, and how to control each one — and its compliance posture is actually the strictest of the major AI platforms. Here is what each agent does, what the fetch volume means, and how to decide what to allow.

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler — its documentation says the bot "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." When ClaudeBot fetches your pages, it is gathering candidate training material, not answering a live user question and not building a search index.

That framing sets expectations correctly. Training crawls reward breadth: the bot fetches widely and repeatedly because corpus building wants coverage and freshness, not because your site is being singled out. In our fleet's logs, ClaudeBot is among the most regular AI visitors across all three production builds, fetching broadly across published pages rather than clustering on any one section [our data].

What are Claude-User and Claude-SearchBot?

They are Anthropic's other two documented agents — a user-triggered fetcher and a search-quality crawler — and each has its own user-agent token. Anthropic's support documentation defines the three-agent set:

Per Anthropic's crawler support documentation, current as of August 2026.

Claude-User is the notable row. User-triggered fetchers are normally the robots.txt gray zone — OpenAI says rules "may not apply" to ChatGPT-User, and Perplexity states Perplexity-User "generally ignores" the file — but Anthropic documents Claude-User as respecting robots.txt directives. Among the platforms in the complete AI crawler list, Anthropic is the only one whose user-fetch agent is documented as declining a page its owner disallowed.

Do Anthropic's crawlers respect robots.txt?

Yes — all three, per Anthropic's documentation, and with one extra courtesy: the Crawl-delay extension. Crawl-delay is a non-standard directive (RFC 9309 does not define it), yet Anthropic documents honoring it, which gives operators a rate-limiting lever most AI crawlers never offer. A group like:

User-agent: ClaudeBot
Crawl-delay: 10

requests a 10-second gap between ClaudeBot fetches — a middle path between open access and a full block for sites where server load, not principle, is the complaint.

Anthropic also supports verification: it publishes its crawler IP addresses in a machine-readable file referenced from the support article, so log entries claiming to be ClaudeBot can be checked against official source IPs. Given how routinely AI user agents are spoofed, filter to verified IPs before drawing any conclusion from "ClaudeBot" lines in a log.

What does ClaudeBot actually fetch — and what does the volume mean?

It fetches a lot and refers a little, and that asymmetry is measured. Cloudflare's crawl-to-click analysis found Anthropic had the steepest crawl-to-refer ratio of any AI platform: roughly 38,066 pages crawled per referral sent in July 2025. The trend line matters as much as the level — that figure was down 86.7% from January 2025's ~286,930:1, the sharpest improvement Cloudflare recorded, but Anthropic still crawled orders of magnitude more than it referred. The metric itself is defined at crawl-to-refer ratio.

Cloudflare's ratio is the measured version of what any allow-all operator should expect: training-crawler fetch volume orders of magnitude above the visits that arrive via Claude-User, the agent that represents an actual human asking about something you published. Our own logs show ClaudeBot as a regular visitor across our builds [our data]; we have not yet published dated per-bot counts from the fleet — when we do, they will be log extracts, not estimates.

The operator takeaway: heavy ClaudeBot traffic is a training-corpus signal, not a citation signal. It tells you Anthropic's corpus builder can reach you — it does not tell you Claude will cite you, and no fetch pattern can promise that.

How do you block or limit ClaudeBot?

One robots.txt group per decision — Anthropic's per-token design means you can block training collection while leaving the other two jobs untouched:

# Block training collection only
User-agent: ClaudeBot
Disallow: /

# Block everything Anthropic documents
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /

Robots.txt remains a compliance request, not a lock; Anthropic documents compliance, and enforcement beyond trust belongs to your CDN or WAF using the published IPs. Anthropic's article also lists a contact address for reporting crawler misbehavior.

Should you block ClaudeBot?

It depends on what your pages earn when a model knows your material — and it is fine for the answer to be "block it." A pageview-monetized publisher sending 38,066 fetches per referral into a training pipeline is subsidizing a system that pays back near nothing; blocking ClaudeBot while allowing Claude-SearchBot and Claude-User is a coherent posture that stops the subsidy without going dark to Claude's users.

We choose the other posture: all three Anthropic agents are allowed across our three production builds [our data], because our sites monetize being known, cited, and visited — not the pageview itself. Assistants that learned our material and can fetch it live are distribution for us. The per-business-model logic is worked through in the robots.txt decision framework, and the wider practice of publishing pages engines can retrieve and quote is our generative engine optimization guide.