Validating Search Console Bulk Export Partitions and Retries
When Search Console sends performance data to BigQuery, it writes each day's data into date-based partitions, which keep each date's rows together.…
When you export performance data from Google Search Console, check which rows each method returns. A report download, an API request, and a BigQuery bulk export follow different rules. A successful download or query does not prove that you received every available row.
The Search Analytics API uses startRow to tell Google which row to start with. It does not use page tokens, which are markers that tell an API where to continue. The API limits the number of rows it returns, and report downloads and bulk exports have their own limits. You can check that your code reached the end of the API results or that BigQuery received the expected daily tables. Neither check proves that Search Console made every possible performance row available.
| Method | How it returns data | Main extraction limit |
|---|---|---|
| Search Console report | Downloads data for the selected report view | The export is limited to representative rows available in the report |
| Search Analytics API | Returns rows in requests controlled by startRow and rowLimit |
Each request allows up to 25,000 rows; API data also has a daily limit |
| BigQuery bulk export | Writes performance data to BigQuery on a daily schedule | It avoids the API’s daily row limit but excludes anonymized queries and depends on successful exports |
Use a report export to get the data shown for a particular view and its filters. Use API pagination to run a query automatically and record each response. Use bulk export to get the broader dataset that Google sends to BigQuery each day. Check each export according to how it delivers data.
A Search Console report export reflects the date range selected for the report, along with its filters and grouping. Google says report exports are limited to 1,000 representative rows, while displayed report totals can include data that does not appear in those rows. The export documentation therefore does not support treating the downloaded row count as a complete inventory of the data behind the report.
Before relying on a file, record the report settings and count its rows. If the report totals are higher than the exported rows account for, the difference does not by itself mean an export error. The totals and the representative rows cover different data. A filtered or grouped file covers its selected view, while other possible combinations of Search Console dimensions remain outside its scope.
The Search Analytics API accepts rowLimit values from 1 to 25,000; its default is 1,000. startRow is a zero-based offset, with a default of 0. Google’s API reference documents these controls and says an offset beyond the available results returns an empty response.
Request one batch after another, and record each request’s offset and returned row count. Stop when a response has fewer rows than you requested. If a response has exactly 25,000 rows, request the next offset. A shorter or empty response shows that the returned results have ended.
offset = 0
page_size = 25000
repeat:
rows = query(startRow=offset, rowLimit=page_size)
save rows
log offset, page_size, number_of_rows_received
offset = offset + number_of_rows_received
if number_of_rows_received < page_size:
stop
Set the next offset by adding the number of rows received. This ties the offset to the actual response. Also save all query settings, including the property, dates, search type, dimensions, and filters. You can then check that every request used the intended scope and that the offsets advanced without gaps.
Reaching the end of the API response does not prove that the API returned every existing row. Google sets a limit of 50,000 rows per day, search type, and property. Google also says internal limitations can cause the API to return the top rows instead of all rows. The API export guidance explains the daily limit. Google’s pagination guidance explains how to request another page when a response is full. Treat “pagination complete” as proof that your client received all rows returned for that query, not as proof that it accessed all the underlying data.
Bulk export writes Search Console performance data to BigQuery on a daily schedule instead of returning pages in response to requests. Google says bulk export is not subject to the API’s daily row limit, but it excludes anonymized queries. The bulk export overview explains that distinction.
For each date, check that every expected table arrived. Search Console exports tables separately, so the presence of one table does not show that the other table for that date arrived. The ExportLog table records successful exports, but not failed attempts. Investigate a missing log entry; it does not explain why an export is missing. Compare the dates and tables in BigQuery with Search Console’s export status and error messages.
Google retries failed exports for about a week; after that, it may drop that day’s data while continuing later exports. The export settings show only the most recent attempt, not the full attempt history. Google’s bulk export monitoring guidance describes these failure and retry conditions. For an ongoing pipeline, also check epoch_version and publish_time. These fields help identify updates Google makes to data for a table and date, as Google documents these fields. When an export version changes, replace or recompute the affected dates instead of simply appending the updated rows.
A BigQuery row count is not necessarily a count of unique dimension combinations. Google advises aggregating exported metrics with functions such as SUM when building totals because rows are not guaranteed to be consolidated by date, URL, or other keys. The BigQuery query guidance provides query recommendations for working with the exported tables.
A completeness check answers two questions: did the method return everything it makes available for this request, and what limits still apply?
For a report export, check the view settings and row count. Keep the report totals separate from the downloaded rows. For an API request, inspect the request log and confirm that each offset advanced by the number of rows returned. Check that the final response was shorter than the requested page size. For bulk export, check that each date and table arrived. Investigate missing log entries or errors, and account for updated table versions.
None of these checks is a universal checksum across Search Console. They establish whether your extraction followed the method’s documented mechanics and reveal where its limits still constrain the data.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com