How to Compare Source Changes Across Repeated AI Answers

When you ask an AI search product the same question more than once, it can cite different web pages in its answers. AI citation drift is a useful name for that change in the sources shown across repeated answers or later checks. It describes what you observed; it does not explain why it happened or tell you whether the answer became less accurate.

To compare answers fairly, keep the prompt and product settings consistent. Record the citations, then check whether each cited page supports the claim beside it. This helps you tell whether only the source list changed or whether the answer’s meaning or reliability changed too.

Why the cited sources change

An AI search product interprets a question, searches for pages, selects information, and writes an answer with links. Changes at any of these steps can change which citations appear. The product usually does not show every page it considered, so a missing citation does not prove that the product never encountered that page.

Some products explain parts of how they search for information and choose sources. Google says AI Overviews and AI Mode may use “query fan-out,” meaning they run multiple related searches for a question. OpenAI says ChatGPT Search may rewrite a question into targeted queries. Those disclosures describe specific products; they do not reveal every retrieval or selection decision behind an individual answer.

Research also finds that generative search results vary across executions and over time. A 2026 comparison of Google search and five generative search systems reported differences in retrieval and synthesis, as well as variation between outputs. The study’s summary does not give one rate of change that applies everywhere. It shows that results vary, but does not predict how often a particular citation will appear.

A source change alone does not establish a change in answer quality. One answer might cite a different page while making the same well-supported claim. Another might keep the same citation while misstating what that page says.

Repeat the prompt under recorded conditions

Save the exact prompt and full answer, along with the date. Also record any citations you can see. Record the product and mode, whether web search was enabled, and any location or language settings the product exposes. If the interface shows a separate source panel, record those links separately from citations attached to answer text.

Repeat the prompt with the same settings. You can use a new chat for each run to avoid carrying earlier messages into the next answer, or keep the same conversation if that is the situation you want to measure. Record which approach you chose. Changing the conversation context changes the conditions of the comparison.

A simple table keeps the evidence together:

Compare exact URLs (full web addresses) and domains (the parts of addresses that identify a website) separately. Comparing domains shows whether the same publisher appears even when the specific page changes. Comparing URLs reveals page changes that domain counts hide. If a link redirects or a page has moved, note the destination rather than treating every URL difference as a different source.

To summarize the results, count how many runs include each URL or domain, then divide by the total number of runs. For example, if a domain appears in three of five runs, report that it appeared in three of five. This describes your sample; it does not estimate how often the product will cite that domain in other questions or conditions.

Keep two questions separate. Repeating the prompt within a short period shows run-to-run variation under the conditions you recorded. Repeating it on later dates shows whether your observations changed over time. The second comparison cannot identify the cause on its own. The product, its search results, the web pages, or your account context might have changed.

Check whether each citation supports the claim

Open each cited page and find the evidence for the statement it seems to support. Check whether the page is relevant, whether it says what the answer claims, and whether its information is current enough for the question. A link beside a sentence does not prove that the page supports it. Some interfaces also show related links that are not direct citations.

Compare the answer’s claims as well as its links. If two answers cite different pages but make the same accurate claim, the citation change may matter little in practice. If the answer changes while its citations remain the same, inspect whether those pages support the new wording.

Whether an answer includes a citation and whether that citation supports the claim are separate questions. In a 2023 evaluation of four generative search products available at the time, researchers found that 74.5% of citations supported their associated sentences. That result applies to the products and methods tested in that study, not to current products generally. It illustrates why readers should verify the evidence rather than treating a citation as a guarantee.

For an important or current claim, use the answer to find relevant evidence, then verify it directly. Prefer a primary source, such as an official document or original study, when it addresses the claim. Several pages repeating one announcement or report do not provide several independent confirmations.

What a changing citation list does not prove

A few repeated answers can show that sources varied in your sample. They cannot establish how often the product cites a source across all answers or prove that it favors, ignores, or increasingly trusts that source. There is no universal number of repetitions that makes every comparison conclusive.

Longer-term comparisons also have limits. The product may not show which model version it uses or every search setting. The systems it uses to find and select pages can also change. Even when you preserve the prompt and visible settings, you may not be able to isolate the cause of a new citation.

A 2025 study uses “citation drift” for changes in references during extended, multi-turn conversations. That is related to this topic, but it is a different measurement setup from repeating one search prompt in separate runs. Define what you are comparing before drawing conclusions from the term.

If you are checking your own website

A successful browser visit or HTTP response (the server’s reply to a web request) shows that the server sent the page in response to that request. It does not show that a particular AI product visited the page, saved it in a search index, found it for a query, cited it, or used its information. Google also explains that meeting its technical requirements does not guarantee that a page will be crawled, indexed, or included in AI features.

For Google-specific visibility, Google said its Search Console generative AI performance reports had rolled out worldwide by August 31, 2026. The reports provide Google Search visibility information; they are not a record of citations across every AI assistant and do not explain why a particular page appeared. Use the relevant product’s reporting where it applies, and keep those measurements separate from repeated manual answer checks.

If your page appears in one answer but not another, first check that people can open it and that its visible text supports the claim you want readers to understand. Then compare the exact prompts, product modes, dates, cited URLs, and answer wording. Treat page access, appearing in search results, being cited, being used accurately, and business results as separate outcomes.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com