Tracking AI visibility over time: know when you drop out of AI answers
Track AI visibility with a fixed prompt set, the same engines and a fixed schedule, scoring mention, citation and accuracy, and watch your AI access too.
LLaunchScaler·Published ·8 min read
Track AI visibility by putting the same fixed set of buyer questions to the same AI engines on a fixed schedule, recording for every answer whether you are mentioned, whether you are cited with a link, and whether you are described accurately, then comparing the rates across runs weeks apart. Check the access side on the same schedule: robots.txt rules, firewall settings and client-side rendering decide whether AI search bots can read your pages at all, and a change there is the fastest way to drop out of answers.
One answer tells you almost nothing. The same question can name different products on two tries, so the signal is in the rates over many answers.
What does AI visibility tracking measure?
AI visibility tracking measures how often, and how accurately, AI engines include you when people ask the questions that lead to buying in your category. Count four things per answer. Mentions and citations show presence, position shows prominence, and accuracy shows whether the answer helps or hurts you when it does name you.
Measure
What you record
Why it matters
Mention
Your product or brand is named in the answer
Presence in the answer at all
Citation
A link to your site appears as a source
The answer can send you a visitor
Position
Where you appear among the products named
Whether you are the recommendation or an afterthought
Questions, answered
What people ask about this
01
How do I track my brand's visibility in AI answers?
Write a fixed set of questions your buyers ask, put them to the same AI engines on a fixed schedule, and record for each answer whether you are mentioned, whether you are cited with a link, and whether the description is accurate. Compare the rates across runs weeks apart rather than reacting to single answers.
Price, features and category are described correctly
A wrong description costs trust even when you are named
Record the competitors named in the same answers too. Your share of mentions against them tells you more than your own count, because it separates "we fell" from "the whole category's answers changed." The metrics are defined in more detail in how to measure AI visibility.
How do you build a repeatable prompt panel?
A prompt panel is a fixed list of questions, a fixed list of engines and a fixed way of asking, run on a schedule. Keeping all three constant is what makes two runs comparable. Change the questions or the setup between runs and a difference could come from the change rather than from the engines.
Write 10 questions a buyer asks before they know your name: "best X for Y," "X alternatives," "how do I solve [the problem you solve]," and one or two comparisons with competitors they already know.
Pick the engines your buyers use, for example ChatGPT, Perplexity, Claude, Gemini, Copilot, Google AI Overviews and Google AI Mode, and keep the list fixed.
Fix the setup: the same account state (signed out, or a clean account with no memory or custom instructions), the same country, the same language, and the default model for each engine.
Ask each question in a new conversation, so earlier answers do not shape later ones.
Record every answer in a sheet with the date, engine, question, and the four measures above, plus the competitors named and the sources cited.
Repeat every two to four weeks on the same weekday.
Ten questions across seven engines gives 70 answers per run, enough to see a rate move. Tools that automate parts of this are compared in the best GEO and AEO tools.
How do you read answer-to-answer noise?
Treat each run as a sample and compare rates, not answers. Engines generate every answer fresh and often search again, so a product can appear in one answer and vanish from the next with nothing having changed. A drop means your mention or citation rate falls across the panel and stays lower on the next run.
Rules that keep noise from driving decisions:
Compare the same engine with itself over time. Engines differ a lot from each other, so a panel-wide average hides an engine-specific drop.
Look at the count, not the percentage alone. Going from 12 to 9 mentions out of 70 answers can easily be ordinary variation; going from 12 to 2 is a different story.
Wait for a second run before acting on a fall, unless you can see an access change that explains it.
Read the sources the engine cited when you were dropped. If a new review page, a competitor's comparison page or a Reddit thread now fills your place, that is the lead to follow.
Note engine and model changes. An engine switching its default model can move every product's rates at once.
What makes a site drop out of AI answers?
The causes split into access and competition. Access failures stop the engine's search bot reading your pages, so you disappear from answers that depend on fresh retrieval. Competition means the bot can still read you, but other sources now answer the question better or more often. Always rule out access first, because it is quick to check and quick to fix.
Cause
Which engines it affects
How to check
robots.txt now disallows a retrieval bot
The engine whose bot is blocked: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, Claude-SearchBot for Claude
Fetch /robots.txt and test each bot against your key URLs
Firewall or bot protection challenges AI bots
Any engine whose bot gets a 403 or a challenge page
curl -A "OAI-SearchBot" -o /dev/null -w '%{http_code}' https://yoursite.com/
Main content moved into client-side JavaScript
Crawlers that read only the raw HTML
curl a page and search for a sentence from its body
Page dropped from Google's index or lost its snippet
Google AI Overviews and AI Mode
URL Inspection in Search Console
The Search generative AI control switched on
Google AI Overviews, AI Mode and Discover's generative features
Search Console, Settings, Search generative AI
Competitors or third-party pages now answer better
Any engine
Compare the sources cited before and after
The bots are separate for a reason. OpenAI says OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features," while GPTBot is "used to crawl content that may be used in training." Perplexity's PerplexityBot is "designed to surface and link websites in search results on Perplexity," and Anthropic's Claude-SearchBot "navigates the web to improve search result quality for users." Blocking the training bots does not have to block the search ones. Both OpenAI and Perplexity say robots.txt changes take about 24 hours to reach their systems, so a bad rule shows up in answers quickly. Monitoring robots.txt for changes covers the file side.
Google is its own case. Its guide to generative AI features says pages must be "indexed and eligible to be shown in Google Search with a snippet," that "you don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and that "seeking inauthentic 'mentions' across the web isn't as helpful as it might seem." For Google's AI features, ordinary indexability is the access check. Other engines publish their own rules, so say which engine a fix is for.
What can you measure in Google's and your analytics tools?
Two first-party reports added in 2026 give you numbers to put beside your prompt panel. Search Console now counts how often your pages appear in Google's AI features, and GA4 now separates visits referred by AI assistants. Neither measures ChatGPT or Perplexity answers directly, but together they show whether AI exposure and AI referrals are rising or falling.
Search Console's Generative AI performance report (Search), rolled out to all sites as of August 31, 2026, counts "how many times links to your site were shown to a user in a generative AI feature on Google Search," for AI Overviews and AI Mode, by page, country, date and device. It reports impressions only.
GA4 added an AI Assistant channel to the default channel group on May 13, 2026. When the referrer "matches a recognized AI Assistant," such as ChatGPT, Gemini or Claude, GA4 assigns the medium ai-assistant and the campaign (ai-assistant).
Review both monthly next to your panel results. A fall in AI Assistant sessions after a fall in your ChatGPT citation rate confirms the drop from a second angle.
How often should you check AI visibility?
Check access on every deploy and at least every two weeks, run the prompt panel every two to four weeks, and review the Search Console and GA4 numbers monthly. Access can break in a minute, and the engines pick up robots.txt changes within about a day; answer rates move slowly and need repeated runs to read.
What
How often
Alert when
robots.txt rules for retrieval bots
Every deploy, and every two weeks
A retrieval bot is newly disallowed
Firewall response to AI bot user agents
Every two weeks, and after firewall or CDN changes
A 403 or challenge page
Raw HTML contains the main content
Every deploy
Key text missing from the server HTML
Prompt panel
Every two to four weeks
Mention or citation count falls on two runs in a row
Generative AI performance report and GA4 AI Assistant channel
Monthly
A sustained fall
Keep the access side checked every two weeks
The prompt panel is yours to run; the access checks are the part a schedule does better than memory. LaunchScaler Watch re-scans one website every two weeks for $29/mo, including the free scan's checks for AI retrieval bots blocked in robots.txt or by a firewall, content that only appears after JavaScript runs, and agent user agents getting a challenge page. It emails a diff on every run with the evidence behind each change, runs uptime checks on your health endpoint in between, and alerts you when a check drops below the bar. Watch does not put questions to the engines; to read how they answer today, the one-time $19 full audit puts 10 questions to 7 answer engines and reads all 70 answers. Start watching.
02
Why do AI answers change every time I ask?
AI engines generate each answer fresh and often search the web again, so the same question can name different products on two runs. That is why a tracker needs many questions and repeated runs, and why a single missing mention is not evidence of a drop.
03
Can Search Console show AI Overviews visibility?
Yes. The Generative AI performance report (Search), rolled out to all sites as of August 31, 2026, counts impressions of your pages in AI Overviews and AI Mode by page, country, date and device. It reports impressions, not clicks.
04
Does GA4 track traffic from ChatGPT?
Yes. Since May 13, 2026, GA4's default channel group includes an AI Assistant channel. When the referrer is a recognised AI assistant such as ChatGPT, Gemini or Claude, GA4 sets the medium to ai-assistant and the campaign to (ai-assistant).
05
Why did my site stop appearing in ChatGPT or Perplexity answers?
Check access first: a robots.txt rule or firewall setting that now blocks OAI-SearchBot or PerplexityBot, or content that moved into client-side JavaScript. If access is fine, compare the answers with earlier runs to see which sources replaced you.
LaunchScaler Watch, Search Console, Semrush, Ahrefs, Screaming Frog, Little Warden, UptimeRobot and more, compared on what they watch, how often and price.
Search Console's Core Web Vitals report uses 28 days of field data, so a regression shows slowly. Add real-user monitoring and lab checks to see it sooner.
When Google organic traffic drops, check tracking, then indexing, then rankings, then AI Overviews and seasonality. The order, the reports and the tells.
