Should you block AI crawlers? A decision table by bot

Block AI training crawlers if you do not want your content used to train models: it is a legitimate choice and it costs you no citations, because the operators run training and search as separate crawlers. Do not block the AI search crawlers or the agents that fetch pages for users if you want ChatGPT, Claude or Perplexity to recommend you, because blocking those removes your pages from their answers.

So "should I block AI crawlers?" has a different answer for each bot. The table below gives it bot by bot, using each operator's own description of what blocking does.

What are the three kinds of AI crawlers?

AI crawlers fall into three groups by what they do with your page. Training crawlers collect content for future models. Search crawlers index pages so an assistant can find and cite them. User-triggered agents fetch a page in real time because a person asked. Blocking each group costs you something different.

Two of the training names are not crawlers at all. Google says Google-Extended has no user agent of its own and is only read "in a control capacity," and Apple says Applebot-Extended "does not crawl webpages." Both are switches that tell the operator how it may use pages its main crawler already fetched.

What does blocking each AI crawler cost you?

Each operator says in its own documentation what a block does. For training crawlers the cost is inclusion in future training; for search crawlers it is citations; for user agents it is the assistant's ability to read your page on request. The table quotes or summarises each operator and gives the usual choice for a product that wants to be recommended.

Apart from Google-Extended's effect on the Gemini app, every "your call" row is a rights decision with no citation cost. Every "allow" row is a crawler that puts your pages in front of someone asking an assistant about your category.

Is blocking only training bots a legitimate setup?

Yes. The operators built their crawlers so you could do exactly this. OpenAI's documentation gives it as the example of independent settings: a site "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot." Anthropic, Google and Apple describe their training controls as separate from search in the same way.

A training-only block keeps live citations intact because the assistants find pages to cite through their search crawlers and user agents, not through training crawls. It does have a cost, just not a citation one: with web search off, an assistant answers from what it learned in training, and a site that opts out is absent from that. For a young product that is rarely the deciding factor, since the answers that name and link products come from search. The comparison of OAI-SearchBot and GPTBot and the comparison of ClaudeBot and Claude-SearchBot show the exact groups for each operator.

Apple adds a wrinkle worth knowing. Its documentation says Applebot-crawled data "may be used to provide additional context and up-to-date content" when AI models generate answers in Siri and Search, answers that "may include links to sources and websites." Blocking Applebot-Extended does not opt you out of those; Apple says the control for them is the nosnippet meta tag on specific content. Even with both in place, Apple says your content "will remain discoverable through Spotlight, Siri, and Safari." So the Apple equivalent of "no training, yes citations" is a disallow for Applebot-Extended and no nosnippet on the pages you want quoted.

How much do training crawlers crawl compared with search crawlers?

Much more. Cloudflare's 2025 Radar review found that "crawling for model training is responsible for the overwhelming majority of AI crawler traffic, reaching as much as 7-8x search crawling and 32x user action crawling at peak." Blocking training crawlers therefore removes most AI crawl load from your server while leaving citations untouched.

Cloudflare's August 2025 analysis put numbers on the shares. Training drove "nearly 80% of AI bot activity, up from 72% a year ago," while search made up 18% and user actions 2% over the same twelve months. Between July 2024 and July 2025, search's share fell from 26% to 17%.

The traffic coming back is lopsided too. The same Radar review found Anthropic's crawl-to-refer ratio reached as much as 500,000 to 1 during 2025, OpenAI's as much as 3,700 to 1 in March, and Perplexity's generally stayed below 400 to 1. If server load from AI bots is your concern, the training crawlers are where it comes from.

When should you block AI search bots or user agents?

Block them only when you do not want an assistant to quote or read the content at all. Licensed or paywalled content, a publication whose business depends on people visiting, or legal limits on redistribution are the usual reasons. For a product site that wants buyers to find it, blocking them trades away the answers that could recommend you.

If the reason is a private area of the site, robots.txt is the wrong tool. RFC 9309 says robots.txt rules "are not a form of access authorization," and Cloudflare's documentation adds that compliance "is voluntary." Two operators say their user-triggered agents may not follow robots.txt at all. Anything that must stay private belongs behind a login.

What does a sensible robots.txt look like?

Write one group that disallows the training tokens you have chosen to block, and one group that allows the search crawlers and user agents with your private paths repeated inside it. A bot that matches a named group ignores your User-agent: * group, so rules you want everyone to follow must appear in each group.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /app/
Disallow: /account/

Leave Google-Extended out of the first group if you want the Gemini app to keep grounding answers on your pages, since Google's token covers that as well as training. The robots.txt guide for AI bots explains each line and the mistakes that silently break it.

Can you block AI search crawlers without meaning to?

Yes, and the cause can sit outside robots.txt. A User-agent: * rule with Disallow: / catches every bot that lacks its own group. Firewall and CDN settings answer before robots.txt is read at all. Check both before assuming your policy is what you wrote.

Cloudflare is worth checking closely in 2026. Its documentation says that from September 15, 2026, new domains block bots classified as Training or Agent on pages that display ads, and that "mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training." Its managed robots.txt, by contrast, disallows training tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot and leaves search crawlers alone. The guide to Cloudflare blocking AI crawlers walks through each setting.

Check what your site actually allows

What you meant to allow and what bots receive can differ, because of a wildcard rule, a CDN setting or a page that only renders with JavaScript. Run the free scan on LaunchScaler: it takes a URL, needs no account and runs 156 checks across 6 of its 7 categories at no cost. Its AI visibility checks fail a robots.txt that blocks OAI-SearchBot, Claude-SearchBot or PerplexityBot, fail a wildcard rule that catches them, pass a training-only block as a legitimate configuration, and test whether a firewall challenges retrieval bots or serves ChatGPT-User and Claude-User a challenge instead of your page.