How to Inspect Identifiable AI Referrals in Website Analytics
Website analytics can identify a visit from an AI service when the visit includes a useful source signal, such as a referring…
Categories
What Raw Data to Request Before Trusting an AI Visibility Dashboard
If you use an AI visibility dashboard to track whether AI products mention or cite your brand, request the records behind its charts before you use its score to judge progress. Most dashboards collect answers to selected questions, then count how often the answers mention or cite a brand, along with other signals. Those observations describe what happened in the dashboard’s measured sample. They do not, on their own, show how often real users saw your brand or whether the dashboard’s results caused business gains.
The most useful AI visibility tool data export contains each question, each individual answer, the product and settings used to collect it, citations, run status, and the rules behind the reported metrics. These fields let you inspect the evidence and check whether the dashboard’s totals match its records.
What the dashboard measures
An AI visibility tool typically runs a set of tracked questions against selected AI products and summarizes the answers. Each tracked question is a prompt sent to an AI product, such as a product-comparison or advice query. Some tools let users add prompts directly, while others also offer suggestions or other ways to build a prompt set, as Peec AI’s prompt documentation describes.
The dashboard’s score depends on which prompts it tracks, how it collects answers, and what it counts. For example, Peec defines visibility as the share of analyzed responses that mention a brand. That is Peec’s formula, not a universal definition of visibility; ask each vendor for its own metric definition and calculation.
Prompt sets and data sources also vary. Adobe’s documentation describes an AI Visibility feature based on a Semrush prompt dataset, while other products document tracking prompts selected for a project. Those approaches measure different samples, so their scores should not be compared by label alone. Check how the vendor selected its prompts and whether it claims they represent observed user behavior.
Request records you can inspect
Ask for an export in a format a computer can read, such as CSV or JSON, with stable IDs that link prompts, answer runs, and citations. A single flat file works if it preserves those relationships; separate files are also useful when each citation links back to its answer and prompt. Gumshoe’s export documentation provides one example of records with prompt, response, and citation details connected by IDs. It describes that product’s export, not a standard that every vendor follows.
| Request | What it lets you check |
|---|---|
| Exact prompt text and stable prompt ID | What the tool asked, and whether prompts were edited, added, removed, or duplicated. Request the complete tracked set, including prompts that produced no brand mention. |
| Prompt origin and tags | Whether you supplied the prompt, the vendor suggested it, or it came from another dataset. Ask for available topic, intent, language, persona, and market labels, plus a record of changes. |
| One record per answer run | The date and time of each observation, with a run or answer ID. Individual records reveal repetitions, gaps, and failed runs that an aggregate can hide. |
| Product and collection details | The AI product, model or version when known, collection route, and relevant settings such as region, language, or web-search mode. Ask the vendor to mark unavailable fields as unknown rather than leaving their status ambiguous. |
| Complete answer text | Whether the dashboard correctly classified a mention, recommendation, or sentiment. A label alone does not show how the brand appeared in context. |
| Citation records linked to answers | The URLs shown as citations and, where available, the source data the vendor separately reports as retrieved or consulted. Ask for both the original URL and any normalized URL or domain. |
| Run status and error details | Which runs succeeded, failed, were skipped, or returned partial results, and how each status affects the score’s denominator. |
| Metric rules and change history | The formula, denominator, weighting, deduplication rules, and aggregation level for every headline metric. Also request a history of changes to prompts, product coverage, settings, collection methods, and metric definitions. |
A vendor might not provide every field. Find out which records the vendor has, which it withholds or cannot capture, and how those gaps affect its reported figures. Ask whether you can reproduce a dashboard total from the export. If the vendor cannot show the inputs and calculation rules, treat the total as an indicator you cannot independently audit.
Check the export against the dashboard
Choose a date range and export all records for its prompts. Confirm that the file contains prompts with no mention as well as prompts where the brand appeared. Then choose one metric and recalculate it using the vendor’s stated formula. For a mention rate, check which answers count, count the answers that contain a mention, and divide by the vendor’s stated denominator, or total number of answers included in the calculation. Include or exclude failed and partial runs according to the documented rule.
Next, inspect several answers yourself. Check that the brand label matches the answer text, that a cited URL is actually linked to the right answer, and that the tool has not confused your brand with a similarly named company or product. Ask whether “citation” means a link visible in the answer or a source the vendor says it detected elsewhere. Keep those meanings separate in your own reporting.
Compare dates only when the prompt wording and measurement settings are comparable. If the vendor changed the prompt set, product coverage, region, or collection method, mark that change next to the trend. A movement in the score could reflect a changed sample or measurement setup; the score alone does not identify the cause.
Separate mentions, citations, traffic, and access
A brand mention, a visible citation, a retrieved source, a site visit, and a sale are separate observations. One does not prove the next. A dashboard might count an answer that names your business without linking to your site. A vendor’s source field might refer to a URL it detected or processed, rather than a citation visibly shown to a user. Request separate fields and definitions before combining these events into a single visibility claim.
Website analytics offer a different kind of evidence. Google Analytics lets users create custom channel groups for AI-assistant referrals, but referral data captures visits with recognizable referral information, not everyone who saw an AI answer. Some users arrive later through direct visits or other channels. Treat AI referrals as a partial traffic signal, and check conversions in analytics or business records separately.
Google Search Console also answers a different question. Google says traffic from its AI features in Search is included in overall Search Console Performance reporting under Web search. That report can help assess Google Search traffic, but it does not provide a prompt-by-prompt record of answers across AI products. Likewise, a server log showing that a crawler, an automated program that requests web pages, requested a page does not prove that an AI product indexed, selected, cited, or recommended it. OpenAI documents separate crawler roles for search, training, and user-triggered actions, which is why access evidence needs its own interpretation.
Failure cases and limits
A chart shows which observations produced the score only when it includes the prompts and answers behind them, along with the formula used to calculate it. Ask for the underlying records and calculation rules. If the vendor does not provide them, state that the score cannot be reproduced from the available export.
A prompt panel is not automatically a representative sample of audience questions. Its results describe the questions and products the tool measured. You can improve its usefulness by adding prompts based on actual customer questions and separating those from vendor-suggested or dataset-derived prompts. Do not describe the combined sample as market-wide demand unless the vendor provides evidence for that claim.
A single run records one answer at one point in time. Repeated runs make variation visible, but even a stable result across a tracked panel does not establish how all users’ answers behave. Peec’s documentation says active prompts run daily by default, with weekly tracking available on some plans; that schedule is specific to its product and configuration, not a standard for the category.
Finally, an export lets you inspect the results, but it cannot show what caused them. If a score rises after a content change, the export may show that the measured answers changed, but it does not by itself prove that the content change caused the result or improved business outcomes. Keep measured observations and changes made in separate records. Record referral traffic and business results separately as well.
Key Takeaways
- Request answer-level exports: Get the exact prompts and full answers behind each chart, along with dates, product details, citations, and run statuses.
- Recalculate one metric: Check its numerator, denominator, and treatment of failed or missing runs against the vendor’s formula.
- Keep comparisons consistent: Record changes to prompts, products, regions, and collection settings before interpreting a trend.
- Separate the signals: Mentions, citations, crawler requests, visits, and conversions describe different events.
- Label the sample honestly: A tracked prompt panel describes its measured questions; it may not represent all real user queries.
What I'm buildingDelegate tasks. Get software.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com