How To

How to Set an AI Visibility Baseline That Stays Comparable

An AI visibility baseline is only useful if you can rerun it the same way. How to pick prompts, platforms, and sample sizes, and log results.

nonBot AI

nonBot AI

Content Team

November 22, 202611 min read

An AI visibility baseline is a recorded snapshot of how often AI assistants mention your brand, which sources they cite, and how they describe you, measured across a fixed set of buyer prompts with repeated runs. To set one, freeze the prompt set, platforms, and testing conditions, then document them so every later run is comparable.

By the end of this process you will have a versioned prompt set, a run log with every answer recorded, the share of runs in which your brand appears for each prompt and platform, and a methodology document that lets anyone repeat the audit later. Before you start, gather a list of your main competitors, every variant of your brand name (including common misspellings and product names), access to the AI assistants you plan to test, a spreadsheet, and raw buyer language from sales calls, support tickets, and review sites. Budget a few working sessions spread across a week, since spreading the runs out is part of the method.

How do you set an AI visibility baseline?

You set one by fixing what you test and how you test it, sampling each prompt several times on each platform, and logging every answer in the same structure. The steps:

  1. Collect buyer questions: Pull the actual wording prospects use from call transcripts, support tickets, Search Console queries, Reddit threads, and review site category pages.

  2. Build the prompt set: Turn those questions into prompts, tag each one by intent and persona, and freeze the list as version one.

  3. Choose the platforms: Pick the assistants your buyers actually use and decide which interface you will test on each.

  4. Set the testing conditions: Decide account state, location, language, and whether memory or chat history is on, then keep those identical for every run.

  5. Run each prompt repeatedly: Ask every prompt several times per platform in fresh sessions, spread across different days.

  6. Log every answer: Record mentions, position in any list, cited URLs, sentiment, and factual errors for each run in its own row.

  7. Calculate appearance by prompt: For each prompt and platform, count how many runs mentioned you out of the total runs.

  8. Write the methodology document: Capture every decision from the steps above so a colleague could repeat the audit without asking you anything.

Steps 2, 3, 5, and 6 need the most judgment, so the sections below cover them in detail.

How many prompts belong in a baseline prompt set?

Enough to cover every way a buyer might discover, compare, and vet you, which for most brands lands somewhere between a few dozen and a couple hundred prompts. Coverage of intents matters more than the raw count. A set of two hundred variations on "best CRM" tells you less than fifty prompts that span the whole buying journey.

Group prompts into intent types and keep each type visible in your reporting:

  • Discovery: "What payroll software works for a restaurant with tip pooling?"

  • Comparison: "Gusto vs a restaurant-specific payroll tool for a three-location group"

  • Alternatives: "What are alternatives to [competitor] for small restaurants?"

  • Problem-led: "How do I handle payroll for servers who work at two locations?"

  • Brand-direct: "Is [your brand] good for restaurants with seasonal staff?"

Keep brand-direct prompts in their own bucket. If you ask an assistant about your brand by name, it will almost always mention you, so mixing those prompts into a headline number makes visibility look better than it is. The discovery and problem-led prompts show whether assistants bring you up when nobody asked about you.

Write prompts the way buyers talk, with the constraints they mention. Real questions include company size, industry, budget, and the tool they are leaving. "Best project management software" is a search engine habit. "Which project management tool should a ten-person design agency use if we hate Jira" is closer to how people type into ChatGPT, and it produces a different shortlist.

Tag each prompt with an ID, intent type, persona, funnel stage, and whether it names any brand. Those tags let you slice results later without rewriting anything.

Which AI platforms should you test?

Test the assistants your buyers use, and record how each one built its answer, because the platforms draw on different mechanisms. Ask your sales team which tools prospects mention, and check your analytics for referral traffic from domains like chatgpt.com and perplexity.ai, which shows where people are already clicking through from AI answers.

Platform

How it builds answers

What to record beyond mentions

ChatGPT

Draws on training knowledge and runs a live web search when it judges one useful

Whether a search ran, and the linked sources if it did

Gemini

Can ground answers in Google Search results

Sources shown and whether they differ from the organic results

Perplexity

Searches the web by default and lists numbered sources

The full source list, including third-party pages that name you

Claude

Draws on training knowledge and can search the web when that capability is on

Whether a search ran, which changes how you interpret a miss

Microsoft Copilot

Searches the web and cites sources alongside its answer

Linked sources and any shopping or product panels

Google AI Overviews and AI Mode

Built from Google's index, shown within search results, and AI Overviews do not appear for every query

Whether an AI Overview appeared at all, plus the linked sources

The "did it search" column matters more than it looks. When an assistant answers without searching, your presence or absence reflects what the model learned in training, which is the territory of Generative Engine Optimization (GEO): Wikipedia, Reddit, Quora, and publications that repeat the same names over time. When it searches, your presence reflects retrieval, which is Answer Engine Optimization (AEO): whether pages that rank for the question today mention or link to you. Logging this lets you route each gap to the right fix later.

Choose an interface per platform and stick with it. Consumer apps can add web search, memory, and their own instructions, so answers from an API often differ from what a buyer sees in the app. For a baseline that reflects buyer experience, test the consumer interface in a logged-out session or a clean account with memory and history off.

How do you account for response variability?

You account for it by sampling each prompt several times and reporting how often you appear, never a single answer. AI answers are non-deterministic. The same prompt can return a different shortlist an hour later because of sampling randomness, small changes in what a search retrieves, or a quiet model update. A brand does not hold a fixed rank in these answers, so a baseline built on one run per prompt is a coin flip recorded as fact.

Five runs per prompt per platform is a reasonable floor for a first baseline. For the prompts closest to revenue, run more, since extra runs tighten your sense of how often you really appear. Spread runs across at least three different days, at different times, so a single bad retrieval day does not define your starting point.

Record each run as its own row, then summarize. A prompt where you appeared in four of five ChatGPT runs and zero of five Perplexity runs tells you something specific. A prompt where you appeared in two of five runs on every platform tells you that you are on the edge of the shortlist and that small gains in third-party coverage could move it. Position within a list can be logged too, but treat it as noisy. Mention order shifts between runs even when the set of names stays the same.

What should you record for each answer?

Record whether you were mentioned, whether you were cited, how you were described, and whether the description was accurate. Mentions and citations are separate signals. A mention is your brand named in the answer text. A citation is a linked source. Assistants can cite your blog post for a definition without naming you as an option, and they can recommend you by name with no link at all, usually because the recommendation came from training.

For sentiment, write coding rules before you look at any answers, so you do not drift as you go. A simple scale works: positive when the answer recommends you or describes you favorably, neutral when you are listed without judgment, negative when the answer warns against you or leads with drawbacks, and mixed when it does both. Paste the exact phrase that drove the code into the log. "Good for small teams but limited reporting" is more useful in six months than the word "mixed".

Add an accuracy flag. If an assistant states the wrong pricing model, names a discontinued product, or confuses you with a similarly named company, record the claim verbatim. Those errors are often fixable at the source and are worth tracking as their own line.

The baseline worksheet structure

Build the worksheet as one spreadsheet with three tabs: a prompt register, a run log, and a methodology tab. Copy the structure below into Google Sheets or Excel and you have the template.

The prompt register holds one row per prompt with these columns: prompt ID, exact prompt text, intent type, persona, funnel stage, brand named (yes or no), and prompt set version.

The run log holds one row per run:

Column

What to record

Example entry

Run ID

Unique ID for the run

R-0142

Prompt ID

Links back to the prompt register

P-017

Platform and interface

Assistant plus app, web, or API

Perplexity, web app

Run date and time

When the prompt was asked

Logged with time zone

Model shown

Any model name the interface displays

As displayed

Search used

Yes, no, or unclear

Yes

Brand mentioned

Yes or no

Yes

List position

Order among named brands, if a list

Third of five

Competitors mentioned

Every other brand named

Gusto, Toast

Cited URLs

Every linked source

Paste all links

Own domain cited

Yes or no

No

Sentiment

Positive, neutral, negative, mixed

Mixed

Descriptor quote

Exact phrase describing you

"Strong on tips, weak on reporting"

Accuracy issue

Any false claim, verbatim

Wrong pricing model

Full answer

Pasted text or screenshot link

Link to screenshot

The methodology tab is covered in the next section. A summary tab with a pivot of appearance by prompt, intent type, and platform is optional, but build it from the run log so the numbers trace back to raw answers.

Saving the full answer text feels like overkill until you need it. When results shift next quarter, the stored answers let you see whether the shortlist changed, the sources changed, or the description changed.

How do you document the methodology so later comparisons stay valid?

Write down every condition that could change an answer, in enough detail that someone else could repeat the audit without you. A later run is only comparable if the prompts, platforms, and conditions match, or if every difference is recorded. Document at least:

  • The prompt set version and where the frozen copy lives

  • Each platform, the interface used, and any model name displayed

  • Account state, including logged in or out, memory and history settings, and any custom instructions

  • Location, VPN use, and language settings

  • Number of runs per prompt and how runs were spread across days

  • The date range of the audit

  • Brand name variants and competitor list used for coding

  • Sentiment coding rules and who did the coding

Keep a change log at the bottom. Assistants update models and search behavior without notice, so note anything you observe, such as a new model name appearing in the interface or citations suddenly showing up where they did not before. When you add prompts later, keep the original set as a fixed core and report it separately from new additions. Otherwise a rise in visibility might just mean you added easier prompts.

Manual sampling at this scale is slow, which is why many teams automate it. Tools such as nonBot AI run a brand's buyer prompts across ChatGPT, Gemini, Perplexity, Claude, and Google AI Overviews and show where the brand appears and which pages get cited. Whether you automate or not, the methodology document is what keeps the comparison honest.

How can you tell the baseline worked?

A baseline works when it is repeatable and when it points to specific gaps. Test repeatability directly: have a colleague rerun a sample of prompts using only the methodology tab. Their appearance counts will not match yours exactly, because answers vary, but they should land in the same neighborhood. If they come back wildly different, a condition is undocumented, usually account state or location.

Usefulness is the second check. You should be able to name the prompts where you are absent, the platforms where the gap is widest, and the third-party pages assistants cite instead of you. If the baseline only produces one blended number, it will not tell anyone what to work on.

Common ways a baseline goes wrong

Three mistakes break most baselines, and all three are easy to avoid at the start.

  • Editing prompts without versioning: Someone rewords a prompt to sound more natural, and the next quarter's comparison quietly measures a different question. Freeze the text, and treat any rewrite as a new prompt with a new ID.

  • Testing in a personalized session: Running prompts in your own logged-in ChatGPT account, where memory knows you work at the brand, inflates mentions. Use logged-out sessions or clean accounts, and record which.

  • Reporting single runs as results: One screenshot of an assistant recommending you, or ignoring you, gets passed around as the verdict. Report how often you appeared across runs and keep the screenshots as examples, never as the metric.

Before you share the baseline with anyone, pick five prompts from the register and have a colleague rerun them from the methodology tab alone, then fix whatever they had to ask you about.

FAQ

How often should I rerun the baseline?

Quarterly works for most brands, with a monthly rerun of a small core set if you are actively publishing or earning coverage. Rerunning too often mostly captures noise from response variability. Always rerun after a major site change or a big press moment so you can see the effect against a known starting point.

Can I use the API instead of the chat apps to save time?

You can, but label it clearly and do not mix the two in one comparison. API responses often differ from the consumer app because the app may add web search, memory, and its own instructions. If your goal is to see what buyers see, the consumer interface is the closer match.

Should competitors be part of the baseline?

Yes. Log every brand named in each answer so you can see who appears where you do not. Competitor data also gives context for your own numbers, since a prompt where nobody appears consistently is a different problem from one where a single competitor dominates.

What if a platform changes its model between my baseline and the next run?

Record the change in the methodology log and compare anyway, but read the shift with caution. A model update can move results on its own, so look for consistency across platforms before crediting your own work. If visibility rose on every platform, your changes are a more plausible cause than one platform's update.

Does referral traffic from AI tools replace prompt testing?

No, the two measure different things. Referral visits from domains like chatgpt.com and perplexity.ai show people clicking cited links, which only happens when you are cited and the reader clicks. Prompt testing shows mentions without links, sentiment, and the answers where you never appeared, which traffic data cannot reveal.

Measure Your Brand's AI Visibility

See how often AI assistants like ChatGPT and Perplexity recommend your business.

Free analysis • No credit card required

Get Started Free →

About nonBot AI: We help brands optimize their visibility across AI platforms—both retrieval-based and training-based. Our AI Visibility tool tracks your presence across ChatGPT, Perplexity, Claude, and more. If you're ready to build a real AIO strategy, talk to an expert.

Related Articles