Free tool
robots.txt checker
Pick a crawler and enter a page. We read your robots.txt the way RFC 9309 says a crawler must, then show whether that crawler may fetch the page and the exact line that decides it.
Example result
- User-agent: *
- Allow: /
- User-agent: GPTBot
- Disallow: /docs/
- Sitemap: https://example.com/sitemap.xml
- FailGPTBot is blocked from /docs/setup by line 5
Your result shows up here
Allowed or blocked for the crawler you picked, the group and the line that decide it, any crawl-delay, every sitemap the file lists, and the lines crawlers skip.
How it works
Three steps, about ten seconds
Enter a page, pick a crawler
Any public URL. Its path and query are what the rules are matched against.
We read robots.txt
We fetch the file at the root of that site and read up to 500 KiB, the most Google reads.
We find the deciding line
The group for that crawler, then the longest rule that matches the page. Allow wins a tie.
Reference
The robots.txt directives, and who reads them
robots.txt is a plain text file at the root of a site. Each line is a directive, a colon and a value, and anything after a # is a comment. Crawlers skip a line they do not understand.
| Directive | What it does | Example | Read by |
|---|---|---|---|
| User-agent | Starts a group and names the crawler it is for. * is any crawler that no other group names. | User-agent: GPTBot | Every crawler |
| Disallow | A path the crawler must not fetch, matched from the start of the path. An empty Disallow blocks nothing. | Disallow: /admin/ | Every crawler |
| Allow | A path the crawler may fetch, used to open a gap inside a Disallow. | Allow: /admin/help | Google, Bing and every crawler that follows RFC 9309 |
| Sitemap | The full address of a sitemap. It belongs to no group, so it can sit anywhere in the file. | Sitemap: https://yourproduct.com/sitemap.xml | Google, Bing and most search crawlers |
| Crawl-delay | Seconds to wait between requests. It is not part of RFC 9309. | Crawl-delay: 10 | Bing. Google ignores it |
The file's own answer matters too. If robots.txt answers with a 4xx, crawlers may fetch anything. If it answers with a 5xx or not at all, RFC 9309 says to treat the whole site as blocked.
Fix it on your platform
Edit robots.txt on your platform
robots.txt has to answer at /robots.txt on the exact host and protocol it covers. Here is where each platform keeps it.
Next.js
Add app/robots.ts. This keeps every crawler out of two folders and names the sitemap:
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
return {
rules: [{ userAgent: "*", allow: "/", disallow: ["/admin/", "/api/"] }],
sitemap: "https://yourproduct.com/sitemap.xml",
};
}A static site or your own server
Put a file named robots.txt in the folder your site serves from, so it answers at /robots.txt with a 200 and text/plain:
User-agent: *
Disallow: /admin/
Sitemap: https://yourproduct.com/sitemap.xmlWordPress
WordPress serves a generated robots.txt until a real file exists in your site's root folder. Yoast SEO edits it under Tools, then File editor. Rank Math edits it under General Settings, then Edit robots.txt, once Advanced Mode is on.
Shopify
Shopify generates robots.txt. To change it, add a template named robots.txt.liquid to your theme's Templates folder. It replaces the generated file.
Webflow
Open Site settings, go to the SEO tab, and write your rules in the robots.txt field under Indexing.
Cloudflare
If your robots.txt holds lines you never wrote, check Cloudflare. Its managed robots.txt setting adds its own lines, including rules for AI crawlers, above yours.
Questions, answered
robots.txt rules, in plain terms
01How does a crawler pick its rules in robots.txt?
- It looks for every group whose User-agent line names it, ignoring case, and merges them. Only if no group names it does it follow the User-agent: * group.
- So a group for Googlebot replaces your * rules for Googlebot. Repeat any * rule you still want inside that group.
02Which rule wins when Allow and Disallow both match?
- The longest rule wins: the one with the most characters in its path. If an Allow and a Disallow are the same length, Allow wins.
- That is RFC 9309, the robots.txt standard, and Google documents the same rule.
03Do * and $ work in robots.txt?
- Yes. * matches any run of characters, and a $ at the end means the URL ends there. Disallow: /*.pdf$ blocks every URL that ends in .pdf.
- Rules match the path and the query, so Disallow: /*?ref= blocks URLs whose query starts with ref=.
04What happens if robots.txt returns a 404 or a 500?
- A 4xx, such as a 404, means there are no rules, so crawlers may fetch anything.
- A 5xx, or no answer at all, means the opposite. RFC 9309 says a crawler must then treat the whole site as blocked. Google treats a 429 the same way as a 5xx.
05Does Google follow Crawl-delay?
- No. Google ignores Crawl-delay. Bing reads it as the number of seconds to wait between requests.
- Googlebot slows down by itself when your server answers with 429, 500 or 503.
06Is there a size limit for robots.txt?
- Google reads the first 500 KiB and ignores the rest. RFC 9309 asks every crawler to read at least that much.
- This checker reads the same 500 KiB and tells you when a file is longer.
07Does blocking a page in robots.txt keep it out of Google?
- No. robots.txt stops crawling, not indexing. Google can still list a blocked URL it finds through links, without a description.
- To keep a page out of results, allow it in robots.txt and add a noindex tag, so Google can read the tag.
08Is it free, and do you keep my results?
- It is free and needs no account.
- A result stays in memory for a few minutes so a repeat check does not hit your site again, then it is gone. Nothing is saved to a database or published.
More free tools
- AI crawler checkerWhich AI crawlers your robots.txt lets in, and what each one gets when it asks for your page.
- ChatGPT search checkerWhether OAI-SearchBot and Bingbot can reach your page, and whether the page lets them show it.
- llms.txt validatorChecks your /llms.txt against the llmstxt.org format, or drafts one from your homepage and sitemap.
- Indexability checkerStatus, redirects, canonical, robots meta, X-Robots-Tag and robots.txt for one URL, as Googlebot reads it.
- Meta tag checkerYour title and meta description as a results page prints them, with the lengths that get cut.
- Open Graph previewYour og: and twitter: tags, the image they point at, and how the link card looks in five apps.
- SSL checkerWhether browsers trust your certificate, who issued it, the names on it and how many days it has left.
- Security headers checkerThe security headers your homepage sends, from CSP to HSTS, and whether plain HTTP redirects to HTTPS.
- Email authentication checkerYour domain's SPF, DKIM and DMARC records, read from DNS, and what each one tells a receiving inbox.