All essays
The Manifest
November 20, 202610 min read

AI Visibility Tracking: When to Act on the Data

AI visibility tracking should help you decide whether something needs attention. That gets difficult when the evidence arrives as a screenshot in Slack: a buyer question, a competitor in the answer, and your company missing.

Imagine that screenshot lands before your marketing meeting. It deserves a closer look, but I wouldn't reorganize the content calendar around it. You don't yet know whether you've found a recurring gap, a different search mode, or one answer that changes when you ask again.

The useful job is building a tracking process that tells you what changed, whether the comparison holds, and who should investigate. Here's how I'd set that up before buying more software or assigning more content.

TL;DR & Takeaways

  • Track a fixed panel of buyer questions repeatedly, with separate results for each engine and search mode.
  • Count brand mentions, recommendations, and owned-site citations separately. They answer different business questions.
  • Check capture quality and repeatability before treating a score change as a content problem.
  • Use a short decision record to connect an observed change to an owner, a bounded action, and a follow-up check.
  • Keep visibility alongside demand and revenue evidence. A sampled answer rate isn't market reach or attributed revenue.

Define what your AI visibility tracking measures

AI visibility tracking records how a brand appears in answers to a defined set of prompts over time. Its usefulness depends on the questions, collection method, and scoring rules. A score describes that sample; it doesn't tell you how often every potential buyer encountered your business.

Before collecting anything, agree on the events you care about.

Signal What counts What it helps you decide
Brand mention The answer names your brand, whether positively, negatively, or neutrally Whether you're present in the tracked conversation
Recommendation The answer presents your offering as an option for the stated need Whether you're being considered for that use case
Owned-site citation The answer links to a source on a domain you own Which of your pages receive source credit
Accuracy issue A factual claim conflicts with a verified current source Whether information needs correction

A third-party review can recommend you without linking to your website. Your article can be cited without the answer recommending your product. Count those events separately, and record third-party sources about your brand in their own field.

I'd count each event once per answer, even if your brand appears repeatedly. Write down accepted brand aliases and owned domains, and keep the sentence supporting each recommendation or accuracy judgment. Ambiguous cases need review rather than an automatic positive score.

This distinction is already established in resources such as Adobe's AI search KPI guide. The harder part is maintaining it when leadership wants one number.

Repeat the questions before explaining the movement

Prefer's September 2026 study asked 80 questions about AI search, GEO, and AEO three times across four engines. In its citation comparison, average pairwise cited-site overlap was 37% for ChatGPT, 35% for Gemini, and 96% for Perplexity.

Those percentages describe Jaccard overlap: sites shared by two answers divided by the distinct sites across both answers. They do not mean ChatGPT retained 37% of a brand's visibility, or that Perplexity will behave that way for your buyers.

Prefer collected the answers on September 13 through API calls, with repeats returning within 23 seconds of one another. The comparison excluded pairs where either answer cited nothing. It measures short-window citation variation in one topic area, not long-term brand performance or the consumer app experience.

Otterly's stability study addresses a different window: daily tracking across three categories. It found brand mentions generally more stable than citations and recommends using citations diagnostically beneath brand coverage.

Both companies sell tracking software. Their studies give you reasons to check your measurement method, not universal thresholds for deciding whether a decline is serious.

I'd begin with three scheduled runs per prompt in a bounded pilot, then repeat that schedule in the next reporting window. Three is a manageable starting rule, not a statistically validated sample size. Important or inconsistent findings need more runs across additional dates.

Keep the counts visible. A recommendation in three of three answers differs from one in two of three, even if both qualify as a majority under your reporting rule. Don't keep rerunning a prompt until the preferred answer appears.

Freeze a buyer-prompt panel without freezing your learning

Build the panel from questions your buyers ask during evaluation, supported by sales conversations, support records, and customer research. Keyword data can suggest topics, but it doesn't prove people use your exact wording in an assistant.

For a hypothetical CRM business, a non-branded prompt might be: “Which CRM suits a small sales team that needs reliable follow-up reminders?” A branded prompt might ask whether that company's CRM supports a specific workflow.

Keep those groups separate. Being named when the question supplies your name tells you something different from appearing in an unaided shortlist.

Give each prompt an ID, an exact wording, a buyer task, and a source for why it matters. Record a panel version. Use a separate exploratory list for new questions so you can learn without quietly changing the denominator in the trend report.

Semrush's reporting guide already recommends fixing the prompt set during a reporting period. I'd add an explicit change log: when prompts change, show a new baseline or compare only an unchanged subset, with its reduced coverage disclosed.

Also record the engine, model when exposed, search mode, language, location, collection time, API or interface, and session setup. Use fresh sessions for the controlled panel. If the model version isn't exposed, record that limitation rather than filling it in from memory.

An API run and a consumer-interface run belong in separate series unless you've established comparability. Personalized conversations can be useful exploration, but they shouldn't quietly replace your controlled checks.

Read the counts before the score

Consider a hypothetical panel of 20 non-branded buyer prompts, each run three times on one engine in each reporting window. Both windows produce 60 valid answers under the same method.

Measure Window A Window B Reading
Answers mentioning the brand 24/60, or 40% 24/60, or 40% Mention rate is unchanged
Answers citing an owned domain 18/60, or 30% 12/60, or 20% Citation rate fell 10 percentage points

You can say owned-site citations declined in this sample. You can't say the brand disappeared from AI search. You also can't infer recommendation performance, because this table hasn't measured it.

Now inspect the prompt-level rows. Did several buyer topics lose citations, or did a few prompts account for the change? Did another page on your domain replace the old one, or did an outside source take its place?

Collection failures complicate this further. Suppose six of the 60 scheduled captures fail and the remaining 54 contain 24 mentions: the valid-answer rate is 44.4%, with capture coverage of 90%.

Reporting only 44.4% would hide the missing observations. If the failed captures cover topics where you rarely appear, the apparent improvement may reflect the changed sample. Recover those captures or show a matched subset before presenting a trend.

A valid answer with no mention belongs in the denominator. A failed capture is missing evidence. A valid answer with no citation is also different from an extraction failure; record whether search occurred when the interface exposes that information.

Report each engine separately before any combined score. If you blend engines or weight buyer topics, disclose the weights and retain the unweighted counts. Don't let a change in collection volume quietly become a change in business importance.

Use a decision record before commissioning work

The practical asset I'd put beside the dashboard is a short decision record. It forces the person reviewing a change to explain why it deserves work, rather than forwarding a red arrow to the content team.

Copy these fields into your existing reporting sheet:

Field What to record
Observation Signal, prompt IDs, engine, windows, and underlying counts
Comparison check Panel and method unchanged? Captures complete? Scoring reviewed?
Repeat evidence Counts by run and date, with links to full answers
Interpretation What the evidence supports, plus a plausible alternative explanation
Decision Rerun, investigate, correct, or hold; name an owner
Follow-up Bounded action, review date, and evidence needed to close the issue

For the hypothetical citation decline, the record could say: “Mentions held at 24/60; owned-site citations fell from 18/60 to 12/60. Review the affected prompt rows and replacement sources before assigning a page rewrite.”

That is a proposed investigation, not a finding that the content became worse. The next observation may support a content change, a technical check, or no intervention.

I'd use these starting rules:

  • One missing citation: rerun on the agreed schedule and preserve the answer. Don't commission content from that observation alone.
  • Recurring absence on a relevant buyer task: inspect cited sources and your existing coverage. Check whether the answer favors a different use case before assuming you need another article.
  • A verified false claim: correct inaccurate owned information promptly and pursue an appropriate correction elsewhere. You don't need weeks of trend data to fix a factual error.
  • A broad shift across many prompts: check the collection method, scoring changes, and platform behavior before treating it as a site-wide content failure.

These are operating rules, not research-derived significance thresholds. Adjust the urgency to the consequence: a wrong eligibility claim can matter more than a larger change in educational citations.

Keep the business question beside the visibility report

The executive question is whether this work helps the business get considered accurately by relevant buyers, and whether that consideration contributes to demand.

Keep AI referrals, qualified inquiries, self-reported discovery, and pipeline evidence beside the sampled visibility data. Record the attribution method and time window. Branded search movement can add context, but other campaigns and market events can move it too.

I wouldn't convert a mention-rate increase into a revenue forecast. I also wouldn't dismiss the work because a buyer discovered the company in an assistant and returned later through another channel. Both conclusions outrun the evidence.

Start with a manual pilot small enough to inspect. Before automating, confirm that someone can reproduce the classifications, identify failed captures, and explain what action the report changes.

Then automate the repeatable collection work, preserve the exceptions for review, and document ownership. If the report never changes a decision, reduce its scope before increasing its frequency.

For your next review, choose the buyer-question panel, name the measurement owner, and complete one decision record against the baseline. The screenshot can start the investigation. The tracking process should determine what happens next.

More from the manifest.

Browse all essays

Ready to grow?

Book a call to discuss your business, industry, and unique opportunities to see how we can help you scale.

Not ready for a call? Reach out directly → hey@makereality.io

Book a discovery call
01Book a time on our calendar that works for you.
02Have a 30 minute call to discuss your business.
03We'll give you some immediate opportunities we see, live on the call that we can carry forward into a project together.