Indexability checkers: tools that show why a page can't be indexed
LaunchScaler, Search Console URL Inspection, Bing URL Inspection, Rich Results Test, Screaming Frog and Sitebulb, compared by what each can tell you.
LLaunchScaler·Published ·9 min read
The indexability checkers worth using are LaunchScaler's free scan, Google Search Console's URL Inspection, Bing Webmaster Tools' URL Inspection, Google's Rich Results Test, and the desktop crawlers Screaming Frog SEO Spider and Sitebulb. They answer different questions: some test whether a page can be indexed, only the search engines' own tools say whether it was, and only crawlers cover a whole site.
They are ranked here for a founder who wants to find out why a page isn't indexable, on any site, without installing anything. Every price and limit was checked on each tool's own pages in September 2026.
What is the difference between indexability and index status?
Indexability is whether a page can be indexed: robots.txt allows crawling, the page returns 200, there's no noindex in the meta tag or X-Robots-Tag header, and its canonical points at itself. Index status is whether Google or Bing actually indexed it. A page can be fully indexable and still not indexed, because the engine chose not to.
The distinction decides which tool you need:
Your question
What answers it
Tools
Can crawlers fetch this page?
robots.txt rules, HTTP status, firewall response
Any checker in this list
Does the page tell engines not to index it?
Meta robots tag and X-Robots-Tag header
LaunchScaler, URL Inspection, crawlers
Which URL does the page say is canonical?
in the head
Questions, answered
What people ask about this
01
What is an indexability checker?
It is a tool that tests whether a page can be indexed: whether robots.txt lets crawlers fetch it, whether it carries noindex in a meta tag or X-Robots-Tag header, whether it returns a 200, and where its canonical points. It does not tell you whether Google actually indexed it.
02
What is the difference between indexability and index status?
LaunchScaler, URL Inspection live test, crawlers with rendering
Did Google index it, and which canonical did Google choose?
Google's index
Search Console URL Inspection only
Did Bing index it?
Bing's index
Bing Webmaster Tools URL Inspection only
Which pages across the whole site are not indexable?
A crawl of every URL
Screaming Frog, Sitebulb
A clean indexability result removes the technical causes. If the page still isn't indexed, the cause is a decision the engine made, such as duplication or quality, and the guide to why some pages aren't indexed walks through that branch.
How do indexability checkers compare?
They differ on three things: whose sites you can test, which directives they read, and whether they report the engine's own index status. The table compares the six on those points.
Tool
Cost
Sites you can test
robots.txt
Meta and header noindex
Canonical
Rendered JS
Index status
Scope
1. LaunchScaler free scan
Free, no account
Any public URL
Yes, per crawler
Both
Yes, including the target's status
Yes
No
The URL, plus the site's robots.txt and sitemap
2. Search Console URL Inspection
Free
Verified properties only
Yes
Both
Declared and Google-selected
Yes, in live test
Yes, for Google
One URL at a time, daily limit
3. Bing URL Inspection
Free
Verified sites only
Reported as an index error
Reported as an index error
Not stated
Live URL fetch
Yes, for Bing
One URL at a time
4. Rich Results Test
Free
Any public URL
Yes
Not reported (a noindexed page can't be tested)
No
Rendered source
No
One URL at a time
5. Screaming Frog SEO Spider
Free to 500 URLs; $279 per licence per year
Any site
Yes
Both
Yes
Yes, integrated Chromium
No
Whole site
6. Sitebulb
14-day free trial; paid plans
Any site
Yes
Both
Yes
Yes, no extra cost
No
Whole site
Which indexability checkers are best?
For most founders the order is LaunchScaler, Search Console URL Inspection, Bing URL Inspection, the Rich Results Test, Screaming Frog, then Sitebulb. LaunchScaler is first because it is the only one that checks every indexability directive on any URL in one pass, with no account and nothing to install. The rest follow from most authoritative for your own site to broadest for a whole-site crawl.
1. LaunchScaler: every indexability directive on any URL, free
LaunchScaler's readiness scan takes any live URL, needs no account, and runs 156 checks across 6 of its 7 categories at no cost: search, AI visibility, security, compliance, speed and visual, and whether the site works. Its search checks cover the indexability directives in one pass.
What it checks for indexability:
robots.txt rules that block a page you want ranked, read per crawler, including the AI retrieval bots OAI-SearchBot, Claude-SearchBot and PerplexityBot
noindex in the meta robots tag or the X-Robots-Tag header
a noindex hidden behind a robots.txt block, which Google can never read
a missing canonical, or one that points at a different URL, a redirect, a 404 or a noindexed page
soft 404s: error or empty pages that return 200
redirect chains that take several hops to reach the final page
main content that only appears after JavaScript runs
invalid HTML in the <head> that drops the canonical and robots tags below it
a missing sitemap, or one that lists redirecting, noindexed or non-canonical URLs
What it can't tell you is whether Google or Bing indexed the page; only their own tools know that. Use it to clear the technical causes, on your site or a competitor's, and then use URL Inspection for the index status.
2. Google Search Console URL Inspection: authoritative, for sites you've verified
URL Inspection is free and is the only tool that reports Google's own index status: whether the URL is on Google, which canonical Google selected, when it was last crawled, and why it wasn't indexed. The catch is scope. The URL must be in a property you have verified, and there is a daily limit on inspections per property.
Its Page indexing section shows "Crawl allowed?", "Page fetch" and "Indexing allowed?", and names the source of a noindex, meta tag or header. Test live URL fetches the current page and, under View tested page, shows the rendered HTML, a screenshot and the HTTP response headers. Canonical choice is only visible in the indexed result; the live test cannot predict it.
One trap to know: when robots.txt blocks a page, "Indexing allowed?" always shows Yes, because Google can't see a noindex behind the block. The URL is not on Google guide reads every line of the panel, and the guide to checking if a page is indexed compares it with a site: search.
3. Bing Webmaster Tools URL Inspection: the same question for Bing and Copilot
Bing's URL Inspection reports whether a URL is in Bing's index, when and where it was discovered, crawled and indexed, with the HTML and HTTP response, plus SEO and markup cards. A Live URL tab shows what Bingbot sees now. It matters beyond Bing search: Bing's guidelines say Bing and Copilot rely on the same crawling and indexing foundation.
It is free and limited to sites you have added and verified in Bing Webmaster Tools. Requesting indexing from the tool draws on your daily URL submission quota. Its Live URL test does not follow redirects; it reports the redirect and its destination, which you then inspect separately.
4. Google Rich Results Test: a crawl check for any public URL
The Rich Results Test is built for structured data, but it fetches any public URL as Google-InspectionTool and reports "Crawl allowed?" and "Page fetch", with the rendered source. That makes it a free way to check whether Google can crawl a page on a site you don't own.
Its limits are the reason it ranks below the tools above. It doesn't report noindex or canonicals; Google's help says a page blocked by robots.txt or noindex simply can't be tested with it. It tests one URL at a time, and it says nothing about index status.
5. Screaming Frog SEO Spider: a desktop crawler for whole-site audits
Screaming Frog's SEO Spider crawls a whole site from your computer and lists every URL blocked by robots.txt, meta robots or X-Robots-Tag, along with canonicals, redirect chains and response codes. It can render pages with its integrated Chromium to crawl JavaScript-heavy sites.
It is free for crawls of up to 500 URLs. A licence removes the limit and adds advanced features; when checked in September 2026, its pricing page listed $279 per licence per year for 1 to 4 licences, with lower per-licence prices for 5 or more, and licences last one year. It reports indexability, not index status, so pair it with Search Console for a site you own.
6. Sitebulb: a desktop crawler with prioritised hints
Sitebulb is also desktop crawling software, for Windows and Mac. Its Indexability Hints cover robots.txt, meta robots, X-Robots-Tag and canonical tags, and it says JavaScript crawling comes at no extra cost. It adds prioritised hints and visual reports on top of the raw crawl data.
Its pricing page lists a Lite plan limited to 10,000 URLs per audit and a Pro plan with a default limit of 500,000 URLs per audit, which can be raised to 2 million on a capable machine, plus a Sitebulb Cloud option. There is a 14-day free trial with no credit card. The desktop plan prices did not appear in the page text we could read, so check them on Sitebulb's pricing page.
Which indexability checker should you use?
Match the tool to the question. For one page on any site, start with a tool that fetches the page itself. For your own site's index status, use the search engines' tools. For a whole site with thousands of URLs, use a crawler.
Your situation
Use
A page you want ranked isn't showing up, and you want the cause fast
LaunchScaler free scan, then URL Inspection
You need to know whether Google indexed a URL and which canonical it chose
Search Console URL Inspection
You care about Bing and Microsoft Copilot
Bing Webmaster Tools URL Inspection
You're checking a competitor's or client's page you can't verify
LaunchScaler free scan, Rich Results Test
A template change may have noindexed hundreds of pages
Whichever you use, confirm fixes where they count: after changing a directive, run Test live URL in Search Console on one affected page before requesting indexing, because Google only reads a noindex or canonical when it next crawls the page.
Check a page's indexability now
Run the free scan on the page that isn't indexing. It takes the address alone, needs no account, and reads the robots.txt rules, both forms of noindex, the canonical and its target, redirects, soft 404s and JavaScript-only content in one pass, so you know which directive to fix before you ask Google to crawl again.
Indexability is whether a page can be indexed, based on what it sends to crawlers. Index status is whether Google or Bing did index it. Only the search engines' own tools, such as Search Console URL Inspection, can report index status.
03
How do I check if a page is indexable for free?
Run LaunchScaler's free scan on the URL with no account, or, for a site you have verified, inspect the URL in Google Search Console. Google's Rich Results Test also reports whether robots.txt allows crawling for any public URL.
04
Can I check indexability for a site I don't own?
Yes, with tools that fetch the page themselves: LaunchScaler's free scan, Google's Rich Results Test, and desktop crawlers such as Screaming Frog or Sitebulb. Search Console and Bing Webmaster Tools only inspect URLs on sites you have verified.
05
Is Screaming Frog free?
Its SEO Spider is free for crawls of up to 500 URLs. A paid licence removes that limit and adds advanced features; its pricing page listed $279 per licence per year for 1 to 4 licences when checked in September 2026.
Google indexed the URL from links without reading it, because robots.txt blocks the crawl. Unblock it and add noindex to drop it, or unblock it to rank it.
