Crawl-to-Refer Ratio

The crawl-to-refer ratio is the number of pages an AI platform's crawlers fetch from websites for every 1 visit that platform's products send back. A ratio of 5:1 means 5 fetches per referred visit; a ratio of 38,000:1 means the platform consumes 38,000 pages for every visitor it returns. Cloudflare introduced the metric to quantify the exchange between AI companies and the sites they crawl — and it has become the standard number in the block-or-allow debate.

How is the crawl-to-refer ratio calculated?

Divide crawler fetches by referred visits over the same period. Cloudflare computes it network-wide: requests from a platform's crawling bots across sites on Cloudflare's network, divided by the visits those platforms refer out, with a high ratio meaning "heavy crawling but little referral traffic" (Cloudflare, August 2025).

The same arithmetic works at single-site scale with 2 numbers you likely already collect. Count requests from a platform's documented crawler user-agents — GPTBot, ClaudeBot, PerplexityBot and peers, verified per the complete AI crawler list — in your server logs. Then count referral sessions from that platform in your analytics, the way our GA4 AI-referral channel does. Fetches over sessions is your ratio.

Stamp the period on both sides of the division. The ratio moves quickly enough that an undated number is close to meaningless.

What are typical crawl-to-refer ratios by platform?

Cloudflare's published snapshot for July 2025, with movement since January 2025 (Cloudflare crawlers blog, August 2025):

Two things stand out. First, the spread spans 4 orders of magnitude — training-dominant crawlers sit at the top, search engines at the bottom. Second, the trend lines cross: Anthropic's ratio fell 86.7% in 6 months while Perplexity's more than tripled from 54:1, so any argument built on a single platform's single-month number is already stale.

In our own fleet's logs, the shape matches the network data: AI crawler fetches vastly outnumber the AI-assistant referral sessions our GA4 channel records [our data]. We publish the method rather than a network-average number, because a 3-brand fleet is a sample of 3, not a benchmark.

Why does the crawl-to-refer ratio matter?

It prices the exchange. Every crawler fetch costs the site bandwidth and compute; every referred visit returns a potential reader or customer. When the ratio is 5:1, crawling looks like the familiar search bargain. When it is 38,000:1, the site is functionally a free data supplier — which is why the metric anchors the publisher block-or-allow decision and the per-bot rules in a robots.txt decision framework.

The ratio is also the honest counterweight to citation enthusiasm. Earning visibility in AI answers is a real goal — it is most of what the operator's guide is about — but the exchange rate on the crawling that feeds those answers is public now, and it is lopsided.

What does the crawl-to-refer ratio miss?

Three things, and they all matter before anyone blocks a bot over a headline number.

It counts visits, not value. Referred visits are the numerator's only counterpart, but answer engines also return brand presence, citations, and assistant-mediated recommendations that never register as a session. A platform can be a poor referrer and still matter to how buyers first hear of you.

It blends crawler jobs. Training fetches, search-index fetches, and live user-triggered fetches serve different products with different payback, and platforms increasingly run separate bots for each. A per-bot log breakdown tells you more than a per-company ratio.

It is a network average. Cloudflare's number describes the whole network's exchange rate, not yours. A site whose content answer engines quote heavily can sit far below the average ratio; a site that never gets cited sits at infinity. Your logs are the ground truth — which is also the cheapest argument against buying a dashboard to learn a number you can compute with grep and a spreadsheet.

How do I measure my own crawl-to-refer ratio?

Four steps, using data you already generate. First, pick a period — 30 days is enough to smooth daily spikes — and hold it fixed on both sides of the division. Second, count the numerator: filter your server or CDN logs to requests from the platform's documented crawler user-agents, using the official strings from each vendor's crawler documentation rather than guesses, since imposters routinely borrow famous bot names. Third, count the denominator: referral sessions from that platform's domains in your analytics — chatgpt.com, perplexity.ai, and peers — which is exactly what a dedicated AI-referral channel group isolates. Fourth, divide, and record the period next to the result.

Two refinements make the number more honest. Split the numerator by bot job where the platform documents separate agents — a training fetch and a live user-triggered fetch are different transactions, and lumping them hides which one drives your costs. And track the ratio as a monthly series rather than a single reading: Cloudflare's own data shows platform ratios moving by 3x or more inside 6 months (January to July 2025), so 1 snapshot ages fast.

On our fleet, the raw inputs are already flowing — CDN logs on the crawl side, a GA4 AI-referral channel on the click side [our data]. Once the filters are written, the computation is an ordinary log query, and it is the only version of the metric that describes your site instead of the internet's average.