Why Changing the Prompt Set Can Create a False Improvement in Visibility

When you measure how often AI search products mention or recommend your business, the questions you test affect the result. A prompt set is the collection of questions used for these checks. If you replace some questions with ones more likely to produce answers about your brand, the reported visibility rate rises even if the answers to the original questions have not improved.

That makes a higher score hard to interpret unless you know which questions were tested and how the score was calculated. Compare results on a stable set of prompts, and report new or changed prompts separately. This lets you tell a like-for-like change from a change in what you measured.

How a new prompt mix raises the score

A visibility rate shows how many tested answers meet a chosen rule. The rule might count an answer that mentions your brand, recommends it, or cites your website. Changing the prompts changes which answers go into the calculation.

For example, a set with more questions that name your brand may produce more brand mentions than a set with more open-ended questions about which products to compare. If you replace some comparison questions with questions that name your brand, the overall rate can rise even though the unchanged questions show no improvement. The score rose because the set contained a different mix of questions. This is a composition effect.

Question wording can also change the answer, even when two prompts seem similar. As of November 11, 2026, Google describes its AI Search features as using related searches to gather information for some responses. OpenAI says ChatGPT Search can rewrite a question into targeted queries and send additional searches after reviewing results. These descriptions show that a prompt can affect what information a product searches for; they do not disclose every retrieval step or mean every prompt triggers the same behavior. See Google’s guidance on AI Search and OpenAI’s description of ChatGPT Search.

A 2026 working paper on prompt wording and brand visibility reports that prompt format and question type changed measured visibility in its tests. The authors describe the work as exploratory. It does not establish an effect size that applies to every brand and platform, or a correction that works for all of them. The paper’s findings support treating prompt choice as part of the measurement, because different prompts test different opportunities.

Check whether your own score is comparable

Start with the prompt lists from both measurement periods. If a dashboard omits the prompt list or answers, or leaves the scoring rule unstated, you cannot reliably tell whether a reported increase came from better results or different questions.

Use this comparison:

1. Find the shared prompts. Calculate the rate using only questions that appeared in both periods for your closest like-for-like comparison.
2. Review what changed. Identify added, removed, rewritten, or duplicated prompts. Note whether the mix shifted between branded questions, open-ended comparisons, informational questions, or buying-stage questions.
3. Check the scoring rule and run conditions. Confirm whether the measurement counted mentions, recommendations, citations, or another outcome. For each run, record the AI search product or interface, date, language, location, and available model or search settings.
4. Inspect individual answers. Check whether a few new prompts, changed wording, or decisions about which answers count as mentions, recommendations, or citations account for the higher average.
5. Repeat some runs when practical. AI answers can vary between runs. Repetition helps show whether a result holds across samples, but no single repeat count works for every platform, topic, and measure.

Then compare the shared prompts with the new ones. If the shared prompts are flat while the total score rises after the prompt mix changes, the increase is consistent with a composition effect. That finding does not prove your real-world visibility stayed flat; it shows that the aggregate score alone cannot establish an improvement.

Keep mentions, recommendations, and citations separate where possible. A passing mention does not carry the same meaning as a recommendation, and a citation does not show by itself that the answer used the cited page accurately.

Keep trend prompts stable and discover new ones separately

Maintain a versioned core prompt set for comparisons over time. When you find new questions that better reflect what customers ask, test them in a separate discovery set first. If you decide to add them to the core, record the change and compare the old and new sets during an overlap period when possible.

A stable set makes trends easier to interpret, but it can become outdated as customer language and buying questions change. A discovery set helps you find new questions without silently changing what the historical score means. If the measurement goal changes, or you give question categories different weights, label the new setup as a new starting point for comparison rather than presenting its score as a direct continuation.

What this check cannot prove

Matching prompts reduces one source of confusion, but it does not isolate every cause of a changed answer. The same question can produce a different result because the search product, available sources, web content, location, or run settings changed. Record the conditions you can observe, and describe unexplained changes as observations rather than attributing them to a particular cause.

A carefully matched prompt set also does not represent every real customer question. And a rise in sampled mentions or citations does not prove more people saw your brand, visited your site, or became customers. Treat prompt tracking as one measurement of answers under defined conditions, then compare it with search reporting, referral traffic, and business outcomes where those data are available.

Key Takeaways

• Compare the shared prompts first: A like-for-like rate shows whether results changed on questions that appeared in both periods.
• Separate new prompts from the core set: Report their results as a separate group before adding them to a historical trend.
• Record the scoring rule and conditions: The AI search product or interface, date, language, location, and what counts as visibility all affect how you interpret the results.
• Keep outcomes distinct: Mentions, recommendations, citations, traffic, and conversions measure different things.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com