Deduplicating Joins Across Citation, Referral, and CRM Data
Website owners comparing citation reports with referrals and customer relationship management (CRM) outcomes need to keep the three record types separate. A…
Google Search Console can show different clicks and impressions in its Performance report, Search Analytics API responses, and BigQuery bulk export. These sources do not always show the same rows or group data the same way. A total can differ because one view counts a property while another counts individual URLs, an export has reached its row limit, or the sources reflect data from different times.
To compare the sources, use the same property and search type. Match the date range in Pacific Time, filters, and aggregation level. Then compare headline totals separately from detailed rows. A mismatch is a clue to investigate, but it does not identify the cause.
Search Console groups performance data by property or page. Property aggregation counts a property once for a query, even if several URLs from that property appear. Page aggregation counts each URL separately, so its impressions and clicks can exceed property-level totals. Average position and click-through rate (CTR) can also change because the underlying counts differ. Google describes these rules in its Performance report documentation.
The report chart uses property aggregation. Tables grouped by Page or Search appearance can count each URL separately. The API’s aggregationType setting controls how results are grouped, and its response includes a responseAggregationType value. Record that response value instead of assuming the API’s default auto setting matches a particular UI table.
Bulk export also separates these grains. The searchdata_site_impression table is property-aggregated; searchdata_url_impression is URL-aggregated. Compare property totals with the first table and URL-level results with the second. A difference between the two is consistent with Google’s aggregation rules. It does not, on its own, show that an export is missing data or that the API is wrong.
The sum of a report’s visible or exported rows may differ from its headline total. Search Console’s UI export is limited to 1,000 representative rows, even when the report total includes more data. The UI export documentation distinguishes report data from the rows included in an export.
The API returns up to 25,000 rows per request. You can paginate with startRow, but Google documents a ceiling of 50,000 rows per day, per property, per search type. The API also states that it does not guarantee every possible row and returns top rows under internal limits. A response that reaches the documented ceiling is evidence that detail may be capped. Fewer returned rows do not prove the result is complete. See the Search Analytics API reference for the request limits and pagination behavior.
Bulk export is not subject to the API and UI row limits, but its BigQuery rows still need correct aggregation. Google says the exported data is not guaranteed to arrive pre-aggregated by date, URL, or dimension combination. Sum clicks and impressions at the intended grain before comparing them; do not treat raw row counts or a query with a LIMIT clause as complete totals. The bulk-export query guidelines describe how to work with the exported tables.
Google omits anonymized query text to protect users, yet the UI can include that activity in chart totals while omitting it from query rows. A query filter also removes anonymized-query activity from the filtered total. As a result, named-query rows can sum to less than an unfiltered chart.
The bulk-export overview says anonymized queries are excluded, while the table schema and query guidance describe anonymized rows with blank query values and an is_anonymized_query flag. Inspect that flag and the query value rather than assuming the query text is available or that every associated metric is absent. The export does not reveal protected query terms. A difference between query-row sums and overall totals alone cannot establish how much of the gap comes from privacy filtering.
Search Console data is normally available after a delay of about two to three days. The newest UI data can be preliminary and can change over the following hours, as Google notes in its data availability guidance. For a stable comparison, use completed dates and finalized API data. The API’s dataState=final setting returns finalized data; dataState=all can include fresh, incomplete data.
Bulk export runs daily, but its tables do not necessarily finish at the same time. A successful write appears in ExportLog, which records the table, reporting date, version, and publication time. A missing log entry does not prove that Search Console had no data: the log records successful saves, not failed attempts. Google documents these fields and export behavior in its bulk-export table guidelines.
Google can also revise previously exported data after correcting a processing error. The epoch_version value starts at 0 and increments when Google updates a table for a date; publish_time records when that export completed. A changed version supports the conclusion that Google recorded a revision for that table and date. It does not explain every historical discrepancy, so first check for missing exports, different dates, or mismatched aggregation.
Use the following workflow to isolate mismatches without treating every difference as a data defect:
Write down the comparison. Record the exact property identifier, search type, start and end dates, and metrics. Also record the aggregation level, breakdown fields, filters, data state, and extraction time. A domain property and a URL-prefix property cover different sets of URLs. Use the same property in the UI, API, and BigQuery.
Align dates and freshness. Search Console uses Pacific Time for daily reporting dates, and API date bounds are inclusive, which means the API includes both the start and end dates. Use the same date labels in every source. For a stable comparison, omit dataState or set it to final in the API. If you include recent data with all, record that choice and inspect any incomplete-data metadata. In BigQuery, confirm that both relevant tables have been exported for every date and note their publish_time.
Compare unfiltered property totals. In the UI, read the chart total for the matching property and search type. Do not sum Page or Search appearance rows. In the API, request the same dates and search type without page or query breakdowns or filters, and record responseAggregationType. In BigQuery, sum clicks and impressions from searchdata_site_impression for the exact property, search type, and dates. For overall totals, include anonymized-query rows or blank query values when present; exclude them only when the analysis specifically concerns named queries.
Test detail completeness separately. For API detail requests, set rowLimit to 25,000 and increase startRow until a response has fewer rows than requested or is empty. If the combined result reaches 50,000 rows for a day and search type, the details may be capped. For a UI export, check whether you exported the chart or the table; table rows have a limit. In BigQuery, combine repeated rows at the level of detail you want to compare.
Match the grain and recompute derived metrics. Use the site-impression table for property totals and the URL-impression table to analyze individual URLs. Calculate CTR from total clicks divided by total impressions, rather than averaging the CTR values from individual rows. For position, divide the source’s position numerator by impressions, then apply the documented one-based adjustment to BigQuery’s stored position value. Google’s bulk table reference describes the position fields and calculation.
Inspect query privacy and export history. For query analysis, check is_anonymized_query and blank query values, then compare query rows with unfiltered totals only with the understanding that protected activity may be absent from named-query detail. For suspected export gaps or revisions, inspect ExportLog by table and date. Treat a recorded version change as revision evidence; treat a missing log row only as evidence that no successful save is recorded.
No universal percentage difference makes a mismatch acceptable. Report whether headline totals match when the aggregation level and dates are the same. Note whether detail rows hit a cap and whether the data is recent or revised. If these checks do not identify a cause, describe the remaining difference without blaming privacy filtering, an export failure, or a product defect.
| Mismatch | What it supports | What it does not prove |
|---|---|---|
| The chart total exceeds the sum of named-query rows | Privacy filtering or row limits could account for some of the gap. | That privacy filtering is the only cause, or that the chart is wrong. |
| A UI table export has fewer rows than an API or BigQuery result | The UI export’s row limit is a plausible explanation. | That the UI headline total omits activity beyond the exported rows. |
| API detail rows do not sum to the headline total | API caps, top-row behavior, or query and URL detail limitations could explain the difference. | That pagination alone caused it, or that the returned rows are complete. |
| Property totals differ from URL-level totals | Property and page aggregation count performance differently. | That either source is defective. |
| Recent dates disagree across sources | Preliminary UI data, API data state, publication lag, or asynchronous exports could explain the difference. | Permanent data loss or a lasting reporting problem. |
A BigQuery table’s epoch_version increases |
Google recorded an update to that table and date. | That the revision explains every mismatch. |
| Raw BigQuery rows appear inflated | Repeated rows or aggregation at the wrong grain could explain the result. | That the exported table is corrupted. |
| A query-filtered total is lower than an unfiltered chart | The query filter can exclude anonymized-query activity. | The identity or exact amount of protected query activity. |
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com