Tracking AI visibility over time: know when you drop out of AI answers

Track AI visibility by putting the same fixed set of buyer questions to the same AI engines on a fixed schedule, recording for every answer whether you are mentioned, whether you are cited with a link, and whether you are described accurately, then comparing the rates across runs weeks apart. Check the access side on the same schedule: robots.txt rules, firewall settings and client-side rendering decide whether AI search bots can read your pages at all, and a change there is the fastest way to drop out of answers.

One answer tells you almost nothing. The same question can name different products on two tries, so the signal is in the rates over many answers.

What does AI visibility tracking measure?

AI visibility tracking measures how often, and how accurately, AI engines include you when people ask the questions that lead to buying in your category. Count four things per answer. Mentions and citations show presence, position shows prominence, and accuracy shows whether the answer helps or hurts you when it does name you.

Record the competitors named in the same answers too. Your share of mentions against them tells you more than your own count, because it separates "we fell" from "the whole category's answers changed." The metrics are defined in more detail in how to measure AI visibility.

How do you build a repeatable prompt panel?

A prompt panel is a fixed list of questions, a fixed list of engines and a fixed way of asking, run on a schedule. Keeping all three constant is what makes two runs comparable. Change the questions or the setup between runs and a difference could come from the change rather than from the engines.

1. Write 10 questions a buyer asks before they know your name: "best X for Y," "X alternatives," "how do I solve [the problem you solve]," and one or two comparisons with competitors they already know.
2. Pick the engines your buyers use, for example ChatGPT, Perplexity, Claude, Gemini, Copilot, Google AI Overviews and Google AI Mode, and keep the list fixed.
3. Fix the setup: the same account state (signed out, or a clean account with no memory or custom instructions), the same country, the same language, and the default model for each engine.
4. Ask each question in a new conversation, so earlier answers do not shape later ones.
5. Record every answer in a sheet with the date, engine, question, and the four measures above, plus the competitors named and the sources cited.
6. Repeat every two to four weeks on the same weekday.

Ten questions across seven engines gives 70 answers per run, enough to see a rate move. Tools that automate parts of this are compared in the best GEO and AEO tools.

How do you read answer-to-answer noise?

Treat each run as a sample and compare rates, not answers. Engines generate every answer fresh and often search again, so a product can appear in one answer and vanish from the next with nothing having changed. A drop means your mention or citation rate falls across the panel and stays lower on the next run.

Rules that keep noise from driving decisions:

• Compare the same engine with itself over time. Engines differ a lot from each other, so a panel-wide average hides an engine-specific drop.
• Look at the count, not the percentage alone. Going from 12 to 9 mentions out of 70 answers can easily be ordinary variation; going from 12 to 2 is a different story.
• Wait for a second run before acting on a fall, unless you can see an access change that explains it.
• Read the sources the engine cited when you were dropped. If a new review page, a competitor's comparison page or a Reddit thread now fills your place, that is the lead to follow.
• Note engine and model changes. An engine switching its default model can move every product's rates at once.

What makes a site drop out of AI answers?

The causes split into access and competition. Access failures stop the engine's search bot reading your pages, so you disappear from answers that depend on fresh retrieval. Competition means the bot can still read you, but other sources now answer the question better or more often. Always rule out access first, because it is quick to check and quick to fix.

The bots are separate for a reason. OpenAI says OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features," while GPTBot is "used to crawl content that may be used in training." Perplexity's PerplexityBot is "designed to surface and link websites in search results on Perplexity," and Anthropic's Claude-SearchBot "navigates the web to improve search result quality for users." Blocking the training bots does not have to block the search ones. Both OpenAI and Perplexity say robots.txt changes take about 24 hours to reach their systems, so a bad rule shows up in answers quickly. Monitoring robots.txt for changes covers the file side.

Google is its own case. Its guide to generative AI features says pages must be "indexed and eligible to be shown in Google Search with a snippet," that "you don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and that "seeking inauthentic 'mentions' across the web isn't as helpful as it might seem." For Google's AI features, ordinary indexability is the access check. Other engines publish their own rules, so say which engine a fix is for.

What can you measure in Google's and your analytics tools?

Two first-party reports added in 2026 give you numbers to put beside your prompt panel. Search Console now counts how often your pages appear in Google's AI features, and GA4 now separates visits referred by AI assistants. Neither measures ChatGPT or Perplexity answers directly, but together they show whether AI exposure and AI referrals are rising or falling.

• Search Console's Generative AI performance report (Search), rolled out to all sites as of August 31, 2026, counts "how many times links to your site were shown to a user in a generative AI feature on Google Search," for AI Overviews and AI Mode, by page, country, date and device. It reports impressions only.
• GA4 added an AI Assistant channel to the default channel group on May 13, 2026. When the referrer "matches a recognized AI Assistant," such as ChatGPT, Gemini or Claude, GA4 assigns the medium ai-assistant and the campaign (ai-assistant).

Review both monthly next to your panel results. A fall in AI Assistant sessions after a fall in your ChatGPT citation rate confirms the drop from a second angle.

How often should you check AI visibility?

Check access on every deploy and at least every two weeks, run the prompt panel every two to four weeks, and review the Search Console and GA4 numbers monthly. Access can break in a minute, and the engines pick up robots.txt changes within about a day; answer rates move slowly and need repeated runs to read.

Keep the access side checked every two weeks

The prompt panel is yours to run; the access checks are the part a schedule does better than memory. LaunchScaler Watch re-scans one website every two weeks for $29/mo, including the free scan's checks for AI retrieval bots blocked in robots.txt or by a firewall, content that only appears after JavaScript runs, and agent user agents getting a challenge page. It emails a diff on every run with the evidence behind each change, runs uptime checks on your health endpoint in between, and alerts you when a check drops below the bar. Watch does not put questions to the engines; to read how they answer today, the one-time $19 full audit puts 10 questions to 7 answer engines and reads all 70 answers. Start watching.