Free tool

AI crawler checker

A single robots.txt line or firewall rule can keep your site out of ChatGPT, Claude and Perplexity answers while it looks fine to every visitor. Check 12 AI crawlers in about ten seconds: we read your robots.txt for each one, then request your page as each crawler that visits.

Example result

OAI-SearchBotAllowedGot the page (200)
PerplexityBotAllowedBot check page (403)
GPTBotBlockedGot the page (200)
ClaudeBotBlockedRefused (403)

Your result shows up here

A row for each of 12 AI crawlers: what your robots.txt says to it and, for the 10 that visit sites, what your server sent back when we asked as that crawler.

How it works

Three steps, about ten seconds

Enter your address

Any public page works. We follow its redirects to the page a crawler would land on.

We read robots.txt

For each crawler token we find the group and the line that decides access, the way RFC 9309 says a crawler reads it.

We ask as each crawler

We request the page with the user agent of each crawler that visits, and compare the answer with what a browser gets.

Reference

The AI crawlers, and the robots.txt line for each

A crawler matches robots.txt on its token, ignoring case. A group that names it replaces the User-agent: * group for that crawler, so a token with its own group ignores your wildcard rules.

AI crawlers, their operators, what each does, and how to block it
Token	Operator	What it does	To block it
OAI-SearchBot	OpenAI	Finds pages to show in ChatGPT search. OpenAI says sites that block it are not shown in ChatGPT search answers.	`User-agent: OAI-SearchBot Disallow: /`
GPTBot	OpenAI	Collects pages that may be used to train OpenAI's models.	`User-agent: GPTBot Disallow: /`
ChatGPT-User	OpenAI	Fetches a page when a ChatGPT user or a custom GPT asks for it. OpenAI says robots.txt rules may not apply.	`User-agent: ChatGPT-User Disallow: /`
Claude-SearchBot	Anthropic	Reads pages to improve the results Claude gives when it searches the web.	`User-agent: Claude-SearchBot Disallow: /`
ClaudeBot	Anthropic	Collects pages that may be used to train Anthropic's models.	`User-agent: ClaudeBot Disallow: /`
Claude-User	Anthropic	Fetches a page when a Claude user asks a question that needs it.	`User-agent: Claude-User Disallow: /`
PerplexityBot	Perplexity	Finds pages to link in Perplexity's answers. Perplexity says it is not used to train AI models.	`User-agent: PerplexityBot Disallow: /`
Perplexity-User	Perplexity	Visits a page when a Perplexity user asks a question. Perplexity says it generally ignores robots.txt.	`User-agent: Perplexity-User Disallow: /`
Bingbot	Microsoft	Crawls pages for Bing. OpenAI names Microsoft among the search providers behind ChatGPT search.	`User-agent: Bingbot Disallow: /`
Google-Extended	Google	Controls whether pages Googlebot crawls may train Gemini and ground its answers. It does not affect Google Search.	`User-agent: Google-Extended Disallow: /`
Applebot-Extended	Apple	Controls whether pages Applebot crawls may train Apple's AI models. Blocked pages can still show in Apple search.	`User-agent: Applebot-Extended Disallow: /`
CCBot	Common Crawl	Builds Common Crawl's open web archive, which many AI models are trained on.	`User-agent: CCBot Disallow: /`

Google-Extended and Applebot-Extended never visit your site. Googlebot and Applebot fetch the pages, and these two tokens only tell Google and Apple whether they may use them for AI training. Operators' pages linked from each token.

Fix it on your platform

Allow or block AI crawlers on your platform

robots.txt decides what a well-behaved crawler asks for, and your firewall decides what it gets. If a crawler is refused while robots.txt allows it, the fix is in the firewall.

Next.js

Add app/robots.ts and give each crawler its own rule. This blocks GPTBot and lets every other crawler in:

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "GPTBot", disallow: "/" },
      { userAgent: "*", allow: "/" },
    ],
  };
}

WordPress

WordPress serves a virtual robots.txt until a real file exists. In Yoast SEO, open Tools, then File editor, which creates or edits a real robots.txt file. In Rank Math, turn on Advanced Mode, then open General Settings, then Edit robots.txt. Rank Math edits the virtual file, so it works only once any real robots.txt is gone from your site's root folder.

Webflow

Webflow adds your sitemap to robots.txt for you. Add your own rules in Site settings, on the SEO tab, in the robots.txt field under Indexing.

Shopify

Shopify generates robots.txt. To change it, open your theme's code editor and add a template named robots.txt.liquid in the Templates folder. It replaces the generated file.

Framer

Framer generates robots.txt. On Pro and Enterprise plans you can serve your own: open Site Settings, then Hosting, then Files, and upload a robots.txt at the root path.

Cloudflare

If the live request shows Refused or Bot check page, look here first. Go to Security, then Settings, filter by Bot traffic, and open AI Crawler and Agent access policies. Search and Agent traffic is what AI answers and assistants send.

Questions, answered

AI crawlers, in plain terms

Which AI crawlers should I allow?

Allow the search crawlers if you want AI answers to cite you: OAI-SearchBot, Claude-SearchBot and PerplexityBot find pages for those answers.
OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers.
Training crawlers (GPTBot, ClaudeBot, CCBot) are a separate choice. Blocking them does not take you out of AI search.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects pages that may be used to train OpenAI's models. ChatGPT search finds pages with OAI-SearchBot.
One robots.txt can block GPTBot and allow OAI-SearchBot. Give each its own User-agent group.

Why is a crawler refused when robots.txt allows it?

robots.txt only asks. A firewall, a CDN's bot setting or a rate limit can still turn the request away.
The live request column shows what your server sent back to each crawler's user agent.
Cloudflare, for example, has a setting that blocks AI crawlers by what they do, separate from robots.txt.

Can I trust a request that says it is GPTBot?

Not on the user agent alone. Anyone can send that string.
OpenAI, Anthropic and Perplexity publish the IP addresses their crawlers use, so you can check a request's address against the list.
It is also why this test can differ from the real crawler: our requests come from our servers, not from theirs.

What if my robots.txt does not load?

A missing file (a 404) means crawlers may fetch anything.
A file that fails with a server error means the opposite. RFC 9309, the robots.txt standard, says a crawler must then treat the whole site as blocked.

How long does a robots.txt change take to apply?

OpenAI says its search systems take about 24 hours to adjust after you change robots.txt.
RFC 9309 asks crawlers not to rely on a cached robots.txt for more than 24 hours.

Is it free, and do you keep my results?

It is free and needs no account.
A result stays in memory for a few minutes so a repeat check does not hit your site again, then it is gone. Nothing is saved to a database or published.

The guide

How to check if AI bots can crawl your site

More free tools

One address, 156 checks, each with its fix.

No account, just the address.

Run the free scan