Why a 300-Run AI Visibility Panel Told Two Stories

Updated 10 November 2026 · Standard formula, shown below

A 300-run AI visibility panel can tell two different stories because the result depends on what was asked, which AI system and answer were sampled, and how “visibility” was counted. Three hundred runs may reduce random noise, but they do not make a panel representative if its prompts, models, timing, or scoring rules differ.

What an AI visibility panel measures

An AI visibility panel is a set of prompts used to observe whether and how a company, product, or source appears in AI-generated answers. A run usually means submitting one prompt to one model or system and recording its response. The panel may report a mention rate, ranking or position, share of answers, citations, sentiment, or some combination of these.

These measures answer different questions. “Mentioned in 40% of runs” is not the same as “recommended first,” and neither necessarily means the system sends visitors to the company. A citation count may also differ from a brand-mention count: an answer can name a business without linking to a source, or cite a page without explicitly naming its publisher in the text.

Before comparing two results, check the panel’s precise metric and denominator. If one report counts any mention and another counts only a first-place recommendation, the apparent disagreement may be definitional rather than evidence that one report is wrong.

Why 300 runs can still produce different results

The number 300 sounds substantial, but the value of a sample depends on how it was assembled. Three hundred nearly identical prompts provide less breadth than 300 prompts covering different needs, phrasings, locations, and stages of a decision. Results can also vary because generative systems do not always return the same answer to the same request.

Prompt mix: “best project-management tools” may invite a different list from “tools for a small nonprofit that needs volunteer scheduling.” If one panel includes more prompts favorable to a category or use case, its overall visibility rate can shift.
Prompt wording: A brand name, competitor, location, or qualification in the question can steer the answer. Even small changes can affect which options appear.
Model and product: Different AI systems, model versions, search settings, and answer modes may rely on different information or produce different responses.
Time and data freshness: Results collected on different dates can reflect changed pages, sources, model behavior, or product settings. Some systems may use web retrieval in a given mode; others may not.
Run conditions: Geography, language, account context, personalization, and whether a fresh conversation is used can all affect what is returned.
Scoring: Rules for aliases, misspellings, citations, subsidiaries, and ambiguous names can change how responses are classified.

For scale, if a simple proportion were based on 300 independent, comparable runs, a measured rate near 50% would have an approximate 95% margin of error of about 6 percentage points under a basic sampling model. That is only a rough illustration, not a guarantee: AI responses may be correlated, prompts are often deliberately varied rather than randomly sampled, and scoring introduces its own uncertainty.

How prompt design changes the story

Prompts are the panel’s equivalent of survey questions. A panel focused on broad category discovery might ask, “What are popular options for…?” A panel focused on comparison might ask, “Compare A and B for….” Those panels are not interchangeable, even if both contain 300 runs. The first measures inclusion in open-ended recommendations; the second measures presence in a constrained comparison.

Prompt selection can introduce bias without anyone intending it. For example, a set with many queries about affordability may favor products known for low prices, while a set emphasizing enterprise integrations may favor a different group. If the panel’s prompts are chosen after seeing early results, the measurements can become especially difficult to interpret.

A useful panel documents its prompt list, explains how prompts were selected, and separates meaningful categories rather than hiding them inside one average. Readers should be able to tell whether a change reflects broad improvement or simply a different mix of questions.

How to compare two panels fairly

Start by treating the two reports as separate measurements, then look for whether their methods match. A credible comparison needs consistent conditions, not just the same total number of runs.

Match the question set. Reuse the same prompts or a documented, balanced sample. Keep wording and language consistent when measuring change over time.
Match the system and settings. Record the AI product, model or mode when identifiable, web-search setting, region, and collection date.
Define a run. State whether repeated submissions of the same prompt count separately, whether conversations are fresh, and how retries or failures are handled.
Fix the scoring rules. Decide in advance what qualifies as a mention, recommendation, citation, or positive answer. Apply the same rules to both datasets.
Inspect the underlying responses. A headline percentage can conceal whether the difference comes from a few prompts, a competitor substitution, or a classification error.
Report uncertainty and coverage. Show counts as well as percentages, break results down by prompt group, and avoid implying that a convenience sample represents every possible user query.

What the result can and cannot tell you

A well-designed panel can help identify patterns: which topics tend to produce a brand mention, which sources are cited, and where answers differ across systems. It can support monitoring when the methodology stays stable. It can also help generate questions for further investigation, such as whether a frequently cited page is accurate and current.

It cannot, by itself, establish how often real people use a particular prompt, prove that an AI system has a fixed ranking, or predict traffic, sales, or market share. A panel’s prompts are observations chosen by its designer, not necessarily a representative sample of actual user behavior. Nor does a mention establish why the model included the name; the answer may be influenced by its training data, retrieved sources, prompt context, or other factors that are not visible in the output.

A common misconception is that a larger run count automatically makes a result objective. More runs can improve precision when the measurement is consistent, but they cannot repair a biased prompt set, mismatched systems, or unclear scoring. Method transparency and repeatability matter as much as the count.

Frequently Asked Questions

Is 300 AI visibility runs enough?
It may be enough to spot directional patterns for a clearly defined panel, but there is no universal threshold. The prompt mix, independence of runs, scoring consistency, and size of the difference all matter.

Why did the same prompt get different AI answers?
Generative systems can produce variable responses, and settings, search results, model updates, conversation context, or personalization may differ. A repeated prompt is not guaranteed to produce identical wording or recommendations.

Should I compare visibility percentages across AI tools?
Only with care. Different tools can use different models, retrieval methods, answer formats, and citation behavior. Compare them as distinct environments and disclose the settings rather than treating the percentages as directly equivalent.

Does an AI mention mean my website was cited?
No. A model can mention a name without linking to its site, and it can cite a page without clearly naming the organization. Track mentions and citations as separate outcomes.

How can I make an AI visibility panel more reliable?
Use a documented, balanced prompt set; keep collection settings stable; define scoring rules before reviewing results; inspect responses; and report breakdowns and counts alongside the headline percentage.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.