ClaudeBot vs Claude-SearchBot vs Claude-User: what each crawler does

Claude-SearchBot is the Anthropic crawler that indexes web pages for Claude's search results; ClaudeBot is the one that collects content for training, and Claude-User fetches a page when someone asks Claude a question that needs it. Each follows its own robots.txt group, so blocking ClaudeBot does not block the other two, and blocking all three takes your own pages out of Claude's answers.

Everything below comes from Anthropic's help article on its crawlers, checked in September 2026, and from the robots.txt standard, RFC 9309.

What is Claude-SearchBot?

Claude-SearchBot is the crawler behind Claude's web search results. Anthropic says it "analyzes online content specifically to enhance the relevance and accuracy of search responses." Blocking it "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results."

In practice, it is the Anthropic crawler that decides whether Claude can find your pages when a user turns web search on. It is the Anthropic equivalent of OpenAI's OAI-SearchBot and Perplexity's PerplexityBot: a crawler whose job is citations, not training. If you want Claude to recommend your product with a link, this is the bot to keep open.

How do ClaudeBot, Claude-SearchBot and Claude-User differ?

They differ by purpose, and Anthropic describes the cost of blocking each one separately. ClaudeBot feeds training, Claude-SearchBot feeds search and Claude-User acts for a user in real time. The table uses Anthropic's own wording.

Anthropic's help article names the tokens but does not publish full user-agent strings. Search your logs for the token itself; under RFC 9309 a crawler's robots.txt token should appear as a substring of its user-agent header.

Does blocking ClaudeBot also block Claude-SearchBot?

No. RFC 9309 says a crawler must find the group that matches its own token and obey that group's rules. A User-agent: ClaudeBot group applies to ClaudeBot only. Claude-SearchBot and Claude-User fall back to your User-agent: * group unless they have groups of their own.

The mix-up runs in both directions. Some sites think they have blocked "Claude" with one ClaudeBot line and are surprised to see Claude-User in their logs. Others think a ClaudeBot block has removed them from Claude search, when it has only opted them out of training.

Cloudflare's managed robots.txt shows the intended pattern. Its generated file disallows ClaudeBot alongside GPTBot, Google-Extended, Applebot-Extended, CCBot and other training crawlers, and does not list Claude-SearchBot or Claude-User, with a content signal of search=yes, ai-train=no. ClaudeBot is also one of the most blocked agents on the web: Cloudflare's 2025 Radar review named GPTBot, ClaudeBot and CCBot as the AI crawlers most often fully disallowed in robots.txt.

What robots.txt groups should you use for each goal?

Pick the goal, then write one group per behaviour. Put the bots you allow in one group with your private-path rules repeated inside it, because a bot that matches a named group ignores the * group. The table maps each goal to its groups and what Claude can still do.

The second row is the one most product sites want:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
Disallow: /app/
Disallow: /account/

To block everything from Anthropic, list all three tokens in one group:

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /

Tokens are matched case-insensitively but must be spelled exactly. claude-searchbot works; Claude-Search-Bot or ClaudeSearchBot matches nothing and leaves that crawler on your * rules. The robots.txt guide for AI bots has a full file with every operator in it.

What happens if you block all three?

Your own pages leave Claude's answers. Claude cannot index them for search, cannot fetch them when a user asks, and will not train on them from then on. Claude can still mention your product from what other websites say about you, if Claude-SearchBot indexed those pages, and from anything collected before you blocked it.

That is a legitimate choice for some sites, such as a publisher that licenses its content or a company with nothing to gain from AI referrals. For a product that wants to be recommended, it is expensive: every time a buyer asks Claude with web search on for tools in your category, your pricing page, docs and comparison pages are out of the pool Claude can quote. The guide on whether to block AI crawlers compares the costs across OpenAI, Anthropic, Perplexity, Google and Apple.

Do Anthropic's crawlers respect robots.txt and Crawl-delay?

Anthropic says yes: "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt." It also supports Crawl-delay as a non-standard extension, so you can slow ClaudeBot down instead of blocking it. This differs from OpenAI and Perplexity, who say their user-triggered fetchers may skip robots.txt.

To throttle rather than block:

User-agent: ClaudeBot
Crawl-delay: 1

Three rules from Anthropic's help article save time:

• Set robots.txt on every subdomain. Anthropic asks you to do this "for every subdomain that you wish to opt out from," so docs.example.com and app.example.com each need their own file.
• Do not rely on IP blocking to opt out. Anthropic warns it "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."
• If a crawler misbehaves, Anthropic lists a contact address in the same article for reports.

The contrast with the other operators is worth knowing when you write rules for all of them. OpenAI's documentation says of ChatGPT-User that "robots.txt rules may not apply," and Perplexity says Perplexity-User "generally ignores robots.txt rules" because a user requested the fetch. Anthropic draws no such exception for Claude-User.

How do Anthropic's crawlers line up with OpenAI's and Perplexity's?

The three operators split their crawlers the same way: one for training, one for search, one for fetches a user triggers. Knowing the equivalents lets you write one consistent policy instead of three. The table pairs them, with what each operator says about robots.txt for the user-triggered fetcher.

A policy of "no training, yes citations" is therefore one Disallow group listing ClaudeBot, GPTBot and any other training crawlers you choose, and one allow group listing Claude-SearchBot, Claude-User, OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User with your private paths repeated inside it. For the two user fetchers that may skip robots.txt, the only hard control is a firewall rule, and blocking them costs you the pages users paste into those assistants.

How do you verify Anthropic's crawlers in your logs?

Check the IP address against Anthropic's published list at claude.com/crawling/bots.json. The file has a creationTime and a prefixes array; in September 2026 it held 26 IPv4 prefixes, created in August 2026. It does not label which prefix belongs to which bot, so it tells you a request came from Anthropic, not which crawler sent it.

1. Pull the requests: grep -E "ClaudeBot|Claude-SearchBot|Claude-User" access.log.
2. Check the status codes each token received. A 200 means it read your page; 403, 429 or a challenge page means a firewall answered first.
3. Check a sample of source addresses against bots.json. A request claiming to be ClaudeBot from outside those prefixes is not Anthropic.
4. Reload bots.json on a schedule if you use it in firewall rules, since the prefixes change.

The AI crawler user agent list has the same verification sources for every other operator, and the guide to getting mentioned by Claude covers what to do once access is right.

Check how your site answers Anthropic's crawlers

A robots.txt that looks right can still fail when a firewall, a CDN setting or a JavaScript-only page gets in the way. Run the free scan on LaunchScaler: it takes a URL, needs no account and runs 156 checks across 6 of its 7 categories at no cost. Its AI visibility checks flag a robots.txt that blocks Claude-SearchBot, OAI-SearchBot or PerplexityBot, fail a wildcard rule that blocks them by accident, treat a training-only block such as ClaudeBot as a legitimate pass, request your page as Claude-User to confirm it gets real content rather than a challenge or an empty shell, and check whether your main content is present in the raw HTML.