Should Publishers Block AI Crawlers? A Decision Framework

"Block the AI bots" and "never block anything" are both sermons, and neither survives contact with a server bill or a citation report. The block-or-allow decision is an economic one, it differs by bot class, and the data to make it now exists. Here is the framework we use, and the choice we made on our own fleet — with the numbers that drove both.

What does the crawl-to-refer data say?

AI platforms take far more pages than they send back visitors, and the imbalance varies by orders of magnitude between platforms. Cloudflare measures this as a crawl-to-refer ratio — HTML pages crawled per page-visit referred — across sites on its network (Cloudflare, 2025):

Even after an 86.7% improvement from January to July 2025, Anthropic still crawled roughly 38,000 pages for every visit it referred — the widest imbalance among major AI platforms. Classic search remains the only crawler class that pays anything like par. The term itself is worth pinning down precisely — our crawl-to-refer ratio glossary entry does — and the wider usage data around it lives on our AI search market share page.

Our own server logs show the same shape at fleet scale: AI crawler fetches run far ahead of the visits those platforms refer back to our 3 builds [our data].

What are the three classes of AI bots?

Every major platform documents distinct agents with distinct jobs, and the decision only makes sense per class:

Sources: OpenAI bot documentation, Anthropic crawler documentation, Perplexity crawler documentation.

Google sits slightly apart: Google-Extended is not a separate crawler but a robots.txt control token that governs whether content Googlebot already fetched can be used for Gemini training — blocking it does not affect Search indexing (Google crawlers overview). The complete inventory, agent by agent, is in our AI crawlers list.

The classes repay you differently. Training crawls almost never return as traffic — that is the 38,000:1. Search-index and user-fetch traffic is the citation path: those fetches happen because an engine is answering, or about to answer, with your page.

When does blocking make sense?

When content is the product rather than the marketing. Three models where blocking training crawlers is economically rational:

• Paywalled or licensed content. If your archive is what customers pay for, letting training crawlers ingest it is uncompensated syndication of the asset itself.
• Ad-monetized libraries. Revenue depends on the pageview; an AI answer that consumes your reporting without the visit strips the impression your model runs on.
• Proprietary data. Original datasets, research, and pricing intelligence — content whose value is exclusivity — lose that exclusivity permanently once ingested.

For these models, the honest configuration is asymmetric: block training crawlers, and decide on search-index bots separately based on whether citations feed anything you monetize. Against our own interest here — we build citation-oriented sites for a living — a paywalled publisher who blocks everything is often making the right call, and nobody selling AI-visibility services should talk them out of it.

When does allowing make sense?

When content exists to generate demand. Lead-generation sites, service businesses, product companies, and anyone whose page is an argument for a purchase gets paid in discovery, not pageviews — and answer engines are now a discovery surface. Blocking search-index and user-fetch bots removes you from those answers while your competitors remain in them, with no compensating revenue on the other side of the ledger.

The realistic expectation matters: Pew found users clicked a source cited in an AI summary on just 1% of encounters (March 2025), so allowing bots buys presence and a thin stream of high-intent clicks, not a traffic channel. That is still a positive trade when the alternative revenue from blocking is zero.

This is our own case. All 3 of our production builds are demand-generation sites, so we allow all three bot classes, and our server logs and referral data are the running record of what that costs and returns [our data].

How do you block without breaking citations?

Per-agent robots.txt rules, backed by CDN enforcement if you are serious. The surgical version disallows training crawlers by user-agent while leaving search-index and user-fetch agents open — every platform above documents its agent names for exactly this purpose. The per-bot decision table, with the records each choice affects, is in our robots.txt decision framework for AI.

Two hard truths about enforcement. First, robots.txt is voluntary: RFC 9309 binds only crawlers that choose to comply (RFC 9309) — the majors document compliance; the long tail does not ask. Second, real enforcement lives at the CDN layer: Cloudflare's managed robots.txt and one-click AI-bot blocking can enforce the policy against non-compliant crawlers at the network edge (Cloudflare). Write the polite version in robots.txt; keep the impolite version available.

What should you actually do this quarter?

Read your own logs before adopting anyone's ideology — including ours. One month of server logs tells you which agents fetch you and how often; your revenue model tells you what a visit is worth; the table above tells you what each class pays back. Most publishers land in one of two stable configurations: block training, allow search and user-fetch (content-as-product), or allow everything and compete for citations (content-as-demand-generation). Both are defensible. Drifting along on defaults you never chose is the only indefensible option.