AI agents fail on challenges, fake buttons, unlabelled fields, hover-only menus and moving layouts. The test and the fix for each, in the order to check.
LLaunchScaler·Published ·8 min read
To make a website usable by AI agents, make sure the agents can reach it, read it and operate it: ChatGPT-User and Claude-User must get a 200 with real HTML rather than a challenge, every control must be a real button or link that works from the keyboard, every form field needs a programmatic label, nothing important may hide behind hover, and the layout must hold still so clicks land where the agent aimed. Each of those has a quick test.
The list below runs in the order an agent hits the problems. A block at the door makes everything after it irrelevant, so start at the top.
What does an AI agent need from a website?
An agent needs to get the page, understand what is on it and act on it. Chrome's Lighthouse documentation says agents "rely on the accessibility tree as their primary data model" and "often rely on screenshots or coordinate-based interaction," which is why labels and stable layouts matter as much as access.
Two kinds of agent visit. Fetchers read a page for a user: OpenAI says ChatGPT-User visits a web page "when users ask ChatGPT or a CustomGPT a question," and Anthropic says Claude-User accesses websites "when individuals ask questions to Claude." Browser agents go further and click, type and submit. Cloudflare's definition of its Agent category covers both: "automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents."
What breaks the session
What the agent experiences
Quick test
Firewall challenge or CAPTCHA
No page at all
curl -I with the agent's user agent; look for 403 or
Questions, answered
What people ask about this
01
How do AI agents browse a website?
They fetch pages for a user, read the page's structure and act on it. Chrome's Lighthouse documentation says agents rely on the accessibility tree as their primary data model, and often use screenshots or coordinates to click, so labels and a stable layout matter.
Load the page in a fresh profile and try to click past it
Can AI agents get past your firewall?
They need a 2xx response with your real HTML, and bot defences can hand them a challenge instead. Anthropic says its bots "respect anti-circumvention technologies" and "will not attempt to bypass CAPTCHAs for the sites we crawl," so a challenge page ends the session. Test both agents before anything else.
Request the page as ChatGPT-User: curl -s -o /dev/null -D - -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://yoursite.com/pricing | grep -iE "^HTTP|cf-mitigated".
Repeat with a user agent containing Claude-User.
A 200 and no cf-mitigated header is a pass. A 403, a 429 or cf-mitigated: challenge means a firewall answered; Cloudflare says that header is present on every challenge page.
Confirm the body is your content, not an empty shell: pipe the same request to grep -c "a sentence from the page". Vercel's crawler study found ChatGPT's agents do not render JavaScript, so a client-rendered page arrives empty.
On Cloudflare, check the AI bot policies under Security Settings. From September 15, 2026, new domains block the Agent behaviour on pages that display ads by default. OpenAI also says robots.txt rules "may not apply" to ChatGPT-User, so if it is being stopped, look at the firewall before robots.txt. The guide to Cloudflare blocking AI crawlers covers each setting.
Your laptop's curl is not the real agent, since it comes from a different IP, so read your firewall's event log for the operator's published addresses to be sure.
Are your buttons and links real controls?
An agent finds controls through the accessibility tree, and a <div> with a click handler is not a control there. WCAG 4.1.2 requires that for every user interface component "the name and role can be programmatically determined," and WCAG 2.1.1 requires that "all functionality of the content is operable through a keyboard interface." Real <button> and <a href> elements meet both by default.
WCAG's own note on 4.1.2 says "standard HTML controls already meet this success criterion when used according to specification." The failures come from custom ones:
Pattern
Problem
Fix
<div onclick="buy()">Buy</div>
No role, not focusable, no keyboard activation
<button type="button">Buy</button>
<a onclick="go()">Pricing</a> with no href
Not a link to software, not reachable by Tab
<a href="/pricing">Pricing</a>
<button><svg>...</svg></button>
A button with no name
Add text or aria-label="Open menu"
A "Start trial" button whose handler failed to attach
Looks fine, does nothing
Test the click, and watch for hydration errors in the console
The test is the keyboard: load the page, press Tab repeatedly and confirm every control you care about receives focus and that Enter or Space activates it. Anything you cannot reach or activate with the keyboard is likely invisible or inert to an agent too. The dead CTA button test covers the last row in detail.
Does every form field have a label?
Every input an agent might fill needs a name it can read. WCAG 3.3.2 requires that "labels or instructions are provided when content requires user input," and WCAG 4.1.2 requires the field's name to be programmatically determinable. Without one, an agent sees an unnamed text box and has to guess what goes in it.
Tie each label to its field with for and id, or wrap the field in the label:
Check the fields agents are most likely to use: signup, contact, demo booking, search and checkout. Lighthouse's agent accessibility tree audit checks the relevant rules, including label, select-name and autocomplete-valid, and lists each failing element. Use a correct autocomplete value too; it tells software exactly what a field expects.
Is anything reachable only on hover?
Content that appears only when a mouse pointer hovers over something is out of reach for a keyboard user, and for an agent that works from the accessibility tree rather than a moving mouse. WCAG 2.1.1 requires all functionality to work from a keyboard, which rules out hover-only menus, and WCAG 1.4.13 sets rules for content that hover or focus does reveal.
Under 1.4.13, content shown on hover or focus must be dismissible without moving the pointer or focus, hoverable so the pointer can move onto it without it vanishing, and persistent until the trigger is removed or the user dismisses it. The usual offenders on product sites are mega menus that open only on mouseenter, pricing details inside tooltips, and "compare plans" rows that expand only on hover.
The fix is to make every hover interaction also work on focus and on click, and to put anything important, such as a price, a limit or a plan difference, in the page as text rather than in a tooltip.
Does the layout hold still long enough to click?
It has to. Chrome's documentation says "unexpected layout shifts can cause an agent to miscalculate the position of a button or input, leading to failed interactions." An agent that locates a button in a screenshot and clicks its coordinates will hit whatever moved into that spot.
web.dev puts a good Cumulative Layout Shift at 0.1 or less, measured at the 75th percentile, and anything above 0.25 is poor. The usual causes are images and embeds without dimensions, ads and banners injected above content, and fonts that reflow text when they swap in. Reserve space for each: width and height on images, aspect-ratio or a minimum height on embed containers, and fixed space for any banner.
Target size helps too. WCAG 2.2's success criterion 2.5.8 asks for pointer targets of at least 24 by 24 CSS pixels, or enough spacing around smaller ones, which also gives a coordinate-based click some margin for error.
How do you test your site the way an agent uses it?
Combine the checks into one pass over the pages that matter: home, pricing, signup, contact and the main product pages. Lighthouse's Agentic Browsing category, shown in PageSpeed Insights, automates two of them, the accessibility tree and layout shift. The rest take a few minutes by hand.
Request each page as ChatGPT-User and as Claude-User; expect 200, no challenge header and your content in the HTML.
Run PageSpeed Insights and read the Agentic Browsing category: the accessibility tree audit and Cumulative Layout Shift.
Tab through each page with the keyboard and confirm every control receives focus and works.
Open every menu and disclosure with the keyboard alone.
Load the page in a fresh browser profile and confirm a cookie banner or modal does not block the controls behind it.
Submit each form once with the keyboard and confirm the success state appears.
Several of these checks need a real browser and several user agents at once. Run the free scan on LaunchScaler: it takes a URL, needs no account and runs 156 checks across 6 of its 7 categories at no cost, and it reports an Agentic Browsing factor out of 3. It requests your pages as ChatGPT-User and Claude-User and fails a challenge or an empty client-rendered shell, checks form labels and link semantics, and loads the page in a browser to find dead calls to action, controls that do not work from the keyboard, cookie banners that block interaction, content exposed only on hover and images that render without reserved dimensions.
The two common causes are a firewall or bot-protection setting that answers ChatGPT-User with a 403 or a challenge page, and a client-rendered page that arrives empty. Request the page with ChatGPT-User's user agent and check for a 200 with your content in the HTML.
03
Do AI agents solve CAPTCHAs?
Anthropic says its bots will not attempt to bypass CAPTCHAs on the sites they crawl. Treat any challenge page as a stop sign for an agent: if it gets one, it gets none of your content.
04
What makes a button usable by an AI agent?
Being a real button or link with an accessible name. WCAG 4.1.2 requires the name and role of every control to be programmatically determinable, and WCAG 2.1.1 requires all functionality to work from a keyboard. A div with a click handler meets neither by default.
05
Is there a score for how usable my site is for AI agents?
Lighthouse has one: its Agentic Browsing category, shown in PageSpeed Insights, reports how many of its agent-readiness audits a page passes, covering the accessibility tree, layout stability, llms.txt and WebMCP.
Measure AI visibility by hand: 10 buyer questions, 7 engines, 4 metrics (mention rate, citation rate, share of voice, accuracy), and how to read the noise.
Search Console now lets you pull a site out of AI Overviews and AI Mode together. Page-level options, what each costs, and why Google-Extended isn't one.
