LaunchScaler

Research

AI crawler readiness of 4,026 new product sites (2026)

AI crawlers get into almost every new product site: only 6 of 4,026 block an AI search bot. The trouble is what they find. 1,319 (32.8%) never name their company in their markup, 354 (8.8%) send under 100 words of HTML before JavaScript runs, and 110 homepages ask search engines not to index them.

Updated ยท Scans from September 14, 2026

Key figures

69.8%

Main content takes longer than 2.5 seconds on a phone

Of the sites with a measured value.

32.8%

No Organization markup on the homepage

8.8%

Fewer than 100 words in the homepage HTML

110

Homepage set to noindex

AI crawlers

32.8%

1,319 of 4,026 sites

No Organization markup on the homepage

Nothing machine-readable on the homepage says which company or brand the site belongs to.

Counted: No JSON-LD node typed Organization, Corporation, Company, Brand, LocalBusiness, OnlineBusiness or another organization type in the homepage HTML (aeo-organization-schema-missing).

Check yours with the free scan

8.8%

354 of 4,026 sites

Fewer than 100 words in the homepage HTML

The page the server sends holds almost no text until JavaScript runs, and that first HTML is what most AI crawlers read.

Counted: Fewer than 100 words of body text in the HTML the server returns for the homepage, before any script runs (search-thin-content).

Check yours with the free scan

0.7%

29 of 4,026 sites

Turn an AI bot away at the server

A browser gets the homepage, but a request that says it is an AI bot gets an error or a challenge page.

Counted: A request with the OAI-SearchBot user agent got an error status or a bot challenge where a browser request got the page (aeo-waf-blocks-retrieval), or the same happened to ChatGPT-User or Claude-User (aeo-agent-ua-access).

Check yours with the AI crawler checker

0.1%

6 of 4,026 sites

Block an AI search bot in robots.txt

Their robots.txt keeps OpenAI, Anthropic or Perplexity's search bot off the homepage.

Counted: robots.txt disallows the homepage for OAI-SearchBot, Claude-SearchBot or PerplexityBot, in a group that names the bot (aeo-retrieval-bots-blocked) or through a User-agent: * rule with no exception for it (aeo-wildcard-blocks-retrieval).

Check yours with the AI crawler checker

Search basics

9.1%

366 of 4,026 sites

Meta description longer than 200 characters

The meta description is too long for a results page to show whole.

Counted: A homepage meta description longer than 200 characters (search-meta-description-overlong). Sites with no meta description at all are not counted here.

Check yours with the Meta tag checker

5.2%

208 of 4,026 sites

Title longer than 70 characters

The homepage title runs past the width a results page shows.

Counted: A homepage <title> longer than 70 characters (search-title-length-truncated).

Check yours with the Meta tag checker

2.7%

110 of 4,026 sites

Homepage set to noindex

The homepage asks search engines to leave it out of their results.

Counted: A noindex or none value in the homepage's meta robots tag or in its X-Robots-Tag header (search-noindex-on-indexable-page).

Check yours with the Indexability checker

2.5%

99 of 4,026 sites

Title missing, empty or a placeholder

The homepage has no usable title, so search engines write one themselves.

Counted: No <title>, an empty one, one that is only a separator and a brand, or a placeholder such as Home, Welcome or Untitled (search-title-missing-or-boilerplate).

Check yours with the Meta tag checker

Speed

69.8%

2,737 of 3,921 sites with a measured value

Main content takes longer than 2.5 seconds on a phone

The largest element on the homepage takes longer than 2.5 seconds to appear on a phone.

Counted: Mobile Largest Contentful Paint above 2.5 seconds, the good threshold of catalog check speed-lcp-field (web.dev). The value is the Chrome UX Report 75th percentile where Google has field data for the site, otherwise the lab value from a mobile PageSpeed Insights run. Counted over the sites with a measured value.

Check yours with the free scan

Security headers

65.5%

2,638 of 4,026 sites

No Content-Security-Policy header

The homepage response sets no Content-Security-Policy.

Counted: No Content-Security-Policy header on the homepage response (sec-csp-missing). Sites that send only a report-only policy are not counted.

Check yours with the free scan

34.8%

1,403 of 4,026 sites

No HSTS header

The HTTPS response never tells browsers to stay on HTTPS.

Counted: No Strict-Transport-Security header on the HTTPS homepage response (sec-hsts-missing).

Check yours with the free scan

Every finding side by side

Share of the 4,026 sites failing each check. Mobile LCP is taken over the 3,921 sites with a measured value.

Share of sites, by finding

  • Mobile LCP over 2.5 s69.8%
  • No CSP header65.5%
  • No HSTS header34.8%
  • No Organization markup32.8%
  • Description over 200 chars9.1%
  • Under 100 words in HTML8.8%
  • Title over 70 characters5.2%
  • Homepage set to noindex2.7%
  • Title missing or generic2.5%
  • AI bot turned away0.7%
  • AI search bot blocked0.1%
Show the data table
Share of sites failing each check
FindingSitesOut ofShare
Main content takes longer than 2.5 seconds on a phone2,7373,92169.8%
No Content-Security-Policy header2,6384,02665.5%
No HSTS header1,4034,02634.8%
No Organization markup on the homepage1,3194,02632.8%
Meta description longer than 200 characters3664,0269.1%
Fewer than 100 words in the homepage HTML3544,0268.8%
Title longer than 70 characters2084,0265.2%
Homepage set to noindex1104,0262.7%
Title missing, empty or a placeholder994,0262.5%
Turn an AI bot away at the server294,0260.7%
Block an AI search bot in robots.txt64,0260.1%

Mobile Largest Contentful Paint

3,921 sites with a measured value, in the web.dev bands. The median is 3.3 seconds.

  • 2.5 s or less30.2%
  • Over 2.5 s, up to 4 s32.7%
  • Over 4 s37.1%
Show the data table
Sites by mobile LCP band
BandSitesShare
2.5 s or less1,18430.2%
Over 2.5 s, up to 4 s1,28132.7%
Over 4 s1,45637.1%

By site category

The category is the one the scan derived from each site's own pages. Sites we could not categorise had the least text in their HTML.

Mobile LCP over 2.5 seconds, by category

  • Marketing77.2%
  • Content & Media75.3%
  • AI74.1%
  • Not categorised70.9%
  • Payments70.3%
  • No-Code69.9%
  • Analytics69.8%
  • Productivity68.7%
  • Design68.6%
  • Developer Tools62.8%
  • Security61.8%
Show the data table
Findings by site category
CategorySitesUnder 100 wordsNo OrganizationNoindexLCP over 2.5 sMedian LCP
Productivity1,0856.1%34.1%2.7%68.7%3.2 s
AI6619.4%28.4%2%74.1%3.5 s
Developer Tools5655%33.3%2.3%62.8%3 s
Analytics3334.8%27.6%1.2%69.8%3.3 s
Content & Media2933.8%25.9%1%75.3%3.5 s
Marketing2444.1%25.4%1.2%77.2%3.7 s
Design19810.1%37.4%6.1%68.6%3.2 s
No-Code1938.3%28%4.1%69.9%3.6 s
Payments1872.7%27.3%0.5%70.3%3.3 s
Security1373.6%34.3%2.2%61.8%2.9 s
Not categorised13088.5%90%16.2%70.9%3.5 s

How this was measured

  • The sample is every website scanned on LaunchScaler between September 14, 2026 and November 9, 2026 whose scan completed: 4,026 sites. Most are products found on public launch directories such as Product Hunt, BetaList, Uneed, Fazier and DevHunt. The rest are sites someone entered into the free scan.
  • Each site counts once, on its most recent scan in that window.
  • A scan fetches the homepage, its robots.txt and its response headers the way a crawler does, repeats the request under AI bot user agents, and runs PageSpeed Insights on a simulated phone.
  • Each finding names the check it counts. The checks are defined in LaunchScaler's audit catalog, each with the published source behind its bar. A site counts when its latest scan failed the check.
  • Left out: 4 sites LaunchScaler runs itself, 27 scans that got no answer from the site, and 0 sites whose owners asked us to remove their data.
  • The figures are counted inside the database and only totals leave it. No site is named here, and the downloads hold the same totals.

Limits

  • A check that could not be read on a site counts as not failing, so every share here is a floor.
  • Only failed checks are stored outside the full report. Results graded as warnings, such as a missing llms.txt, a missing meta description or a missing sitemap, are not in this study.
  • The robots.txt figure covers the search bots that decide whether an AI answer can cite a site. Blocks aimed only at training bots, such as GPTBot, ClaudeBot or CCBot, are not counted.
  • The homepage stands in for the site. Inner pages can differ.
  • Mobile LCP mixes field data and lab runs, depending on what Google holds for each site.
  • These are sites that reached LaunchScaler, mostly from launch directories, not a random sample of the web.

Download the data

The findings file has one row per finding and LCP band, with the count, the sample, the share, the catalog checks and the definition. The category file has one row per site category.

How to cite

Suggested citation
LaunchScaler, "AI crawler readiness of new product websites, November 2026," November 9, 2026. Sample: 4,026 websites scanned September 14, 2026 to November 9, 2026. https://www.launchscaler.com/research/ai-crawler-readiness

Updated . Snapshot file ai-readiness-2026-11.

One address, 156 checks, each with its fix.

No account, just the address.

Run the free scan