Which AI crawlers is your robots.txt blocking?

The crawlers worth knowing

GPTBot (OpenAI), ClaudeBot and Claude-Web (Anthropic), PerplexityBot, Google-Extended (Gemini training and grounding, separate from Googlebot), CCBot (Common Crawl, which feeds many training sets), Bytespider and Applebot-Extended.

Google-Extended is the one that surprises people: blocking it does not affect classic Google Search ranking, but it does affect Gemini. They are separate switches.

The rule that catches everyone

robots.txt group matching is winner-take-all. A crawler uses the most specific group that names it, and falls back to * only if no such group exists. It does not merge them.

So this blocks every AI crawler, even though it never names one:

User-agent: *
Disallow: /

# Nothing below rescues GPTBot — it never gets its own group,
# so it falls back to * and is blocked.

Allowing AI crawlers deliberately

Give each bot an explicit group. Scope the Disallow to what genuinely should not be public rather than blocking the root:

User-agent: GPTBot
Disallow: /admin/
Disallow: /account/

User-agent: ClaudeBot
Disallow: /admin/
Disallow: /account/

User-agent: PerplexityBot
Disallow: /admin/
Disallow: /account/

User-agent: *
Disallow: /admin/
Disallow: /account/

Should you allow them at all?

This is a business decision, not a technical one. Blocking protects content from training use; allowing makes you citable in AI answers. Publishers with paywalled archives often block. Companies that want to be recommended when someone asks an assistant for a vendor generally should not.

What is rarely right is blocking by accident — a Disallow: / copied from a staging config, still live in production, quietly making you invisible.