Robots.txt for AI Crawlers: A Decision Framework

Most robots.txt advice for AI is a sermon — "block the freeloaders" or "never block anything" — delivered without asking what the site sells. The file deserves better: it is a per-bot instrument, every documented AI agent does one of three jobs, and the cost of blocking each job depends entirely on your business model. This is the decision framework we use on our own builds, grounded in RFC 9309 and each platform's crawler documentation.

What does robots.txt actually control?

A robots.txt file publishes crawl rules that compliant crawlers must follow — and nothing forces compliance. The Robots Exclusion Protocol was standardized as RFC 9309 in September 2022, formalizing the convention Martijn Koster started in 1994. Three of its properties shape every AI decision:

It is not security. The RFC states its rules "are not a form of access authorization." A crawler that ignores the file has broken a norm, not a lock — which is why enforcement questions end at the CDN, covered below.

Matching is specific and literal. Rules group under User-agent lines, and "the most specific match found MUST be used." A Disallow for GPTBot says nothing about OAI-SearchBot; every token needs its own decision.

Parsers must handle at least 500 kibibytes. So even a long per-bot file with a group for every documented AI agent costs you nothing technically.

What are the three jobs an AI bot can have?

Training, search indexing, and user-requested fetching — and the job, not the logo, sets the price of blocking. The full token-by-token roster is in the complete AI crawler list; the decision-relevant summary:

Per vendor crawler docs, August 2026.

Google needs its own paragraph because its AI surfaces do not hang off an AI token at all. Eligibility for AI Overviews and AI Mode follows snippet eligibility — the levers are the documented snippet controls nosnippet, data-nosnippet, max-snippet, and noindex, each of which also constrains your classic-result appearance. Google-Extended, meanwhile, is a standalone product token with no user agent of its own: disallowing it governs Gemini training and certain grounding uses, and Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Blocking Google-Extended is a training decision; it will not remove you from AI answers on Google.

How should different business models decide?

Price each job against how the site earns. The framework we apply in audits, one row per model:

Ratios cited are Cloudflare's crawl-to-refer measurements for July 2025; Google measured ~5:1. The crawl-to-refer ratio has its own glossary entry, and the publisher-specific version of this decision gets a full treatment in should publishers block or allow AI.

The one indefensible posture is the unexamined blanket block. Pasting a block-all-AI list blocks OAI-SearchBot alongside GPTBot — trading away documented answer visibility to stop a training crawler, usually without noticing. Every line should name a job you mean to refuse.

What do the actual files look like?

Three coherent postures, written with per-token groups. Ours first, argued honestly:

# Posture 1 — allow everything (what we run)
# No AI-specific groups; absence of Disallow is consent.

# Posture 2 — refuse training, keep answer visibility
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /

# Posture 3 — refuse all documented AI crawling
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /

We run posture 1 across all 3 production builds [our data]: our sites earn from being retrieved, cited, and visited, so search crawlers and user fetchers are distribution, and training access is the toll we accept. Against our own interest: if we operated a subscription publisher, we would run posture 2 without hesitation — the crawl-to-refer math for training bots is a subsidy that model only loses on. The per-vendor details behind these groups are in GPTBot vs OAI-SearchBot and Anthropic's three crawlers.

What if a bot ignores robots.txt?

Then you have left the protocol and entered enforcement, which is a network-layer job. Cloudflare documents CDN-level controls for AI crawlers — including a managed robots.txt and the ability to block identified AI bots at the edge regardless of whether they honor your file. Equivalent WAF rules work on any stack: match the vendor's published IP ranges (OpenAI, Anthropic, and Perplexity all publish theirs) and refuse or rate-limit at the door.

Verify before you convict, though. AI user agents are spoofed constantly, and a "PerplexityBot" from an unlisted IP is somebody else's scraper wearing the name. Check source IPs against the published lists — Google additionally documents reverse-DNS verification — before attributing behavior to a platform, or you will end up enforcing against a ghost. How all of this connects to actually being cited — the reason most sites leave the doors open at all — is the subject of our generative engine optimization guide.