Currently Available: Need a skilled Software Developer for your next project?
Categories
SEO, GEO and AEO

Handling Search Console Export Limits and Pagination Gaps

When you export performance data from Google Search Console, check which rows each method returns. A report download, an API request, and a BigQuery bulk export follow different rules. A successful download or query does not prove that you received every available row.

The Search Analytics API uses startRow to tell Google which row to start with. It does not use page tokens, which are markers that tell an API where to continue. The API limits the number of rows it returns, and report downloads and bulk exports have their own limits. You can check that your code reached the end of the API results or that BigQuery received the expected daily tables. Neither check proves that Search Console made every possible performance row available.

Three exports return data differently

Method How it returns data Main extraction limit
Search Console report Downloads data for the selected report view The export is limited to representative rows available in the report
Search Analytics API Returns rows in requests controlled by startRow and rowLimit Each request allows up to 25,000 rows; API data also has a daily limit
BigQuery bulk export Writes performance data to BigQuery on a daily schedule It avoids the API’s daily row limit but excludes anonymized queries and depends on successful exports

Use a report export to get the data shown for a particular view and its filters. Use API pagination to run a query automatically and record each response. Use bulk export to get the broader dataset that Google sends to BigQuery each day. Check each export according to how it delivers data.

UI exports reflect report limits

A Search Console report export reflects the date range selected for the report, along with its filters and grouping. Google says report exports are limited to 1,000 representative rows, while displayed report totals can include data that does not appear in those rows. The export documentation therefore does not support treating the downloaded row count as a complete inventory of the data behind the report.

Before relying on a file, record the report settings and count its rows. If the report totals are higher than the exported rows account for, the difference does not by itself mean an export error. The totals and the representative rows cover different data. A filtered or grouped file covers its selected view, while other possible combinations of Search Console dimensions remain outside its scope.

API pagination needs a final-page check

The Search Analytics API accepts rowLimit values from 1 to 25,000; its default is 1,000. startRow is a zero-based offset, with a default of 0. Google’s API reference documents these controls and says an offset beyond the available results returns an empty response.

Request one batch after another, and record each request’s offset and returned row count. Stop when a response has fewer rows than you requested. If a response has exactly 25,000 rows, request the next offset. A shorter or empty response shows that the returned results have ended.

offset = 0
page_size = 25000

repeat:
    rows = query(startRow=offset, rowLimit=page_size)
    save rows
    log offset, page_size, number_of_rows_received

    offset = offset + number_of_rows_received

    if number_of_rows_received < page_size:
        stop

Set the next offset by adding the number of rows received. This ties the offset to the actual response. Also save all query settings, including the property, dates, search type, dimensions, and filters. You can then check that every request used the intended scope and that the offsets advanced without gaps.

Reaching the end of the API response does not prove that the API returned every existing row. Google sets a limit of 50,000 rows per day, search type, and property. Google also says internal limitations can cause the API to return the top rows instead of all rows. The API export guidance explains the daily limit. Google’s pagination guidance explains how to request another page when a response is full. Treat “pagination complete” as proof that your client received all rows returned for that query, not as proof that it accessed all the underlying data.

BigQuery exports need date and table checks

Bulk export writes Search Console performance data to BigQuery on a daily schedule instead of returning pages in response to requests. Google says bulk export is not subject to the API’s daily row limit, but it excludes anonymized queries. The bulk export overview explains that distinction.

For each date, check that every expected table arrived. Search Console exports tables separately, so the presence of one table does not show that the other table for that date arrived. The ExportLog table records successful exports, but not failed attempts. Investigate a missing log entry; it does not explain why an export is missing. Compare the dates and tables in BigQuery with Search Console’s export status and error messages.

Google retries failed exports for about a week; after that, it may drop that day’s data while continuing later exports. The export settings show only the most recent attempt, not the full attempt history. Google’s bulk export monitoring guidance describes these failure and retry conditions. For an ongoing pipeline, also check epoch_version and publish_time. These fields help identify updates Google makes to data for a table and date, as Google documents these fields. When an export version changes, replace or recompute the affected dates instead of simply appending the updated rows.

A BigQuery row count is not necessarily a count of unique dimension combinations. Google advises aggregating exported metrics with functions such as SUM when building totals because rows are not guaranteed to be consolidated by date, URL, or other keys. The BigQuery query guidance provides query recommendations for working with the exported tables.

Verify the scope before trusting the numbers

A completeness check answers two questions: did the method return everything it makes available for this request, and what limits still apply?

For a report export, check the view settings and row count. Keep the report totals separate from the downloaded rows. For an API request, inspect the request log and confirm that each offset advanced by the number of rows returned. Check that the final response was shorter than the requested page size. For bulk export, check that each date and table arrived. Investigate missing log entries or errors, and account for updated table versions.

None of these checks is a universal checksum across Search Console. They establish whether your extraction followed the method’s documented mechanics and reveal where its limits still constrain the data.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com

Email updates

Usually a new article and a few links I found interesting.

No spam. Unsubscribe with one click.

Leave a Reply

Your email address will not be published. Required fields are marked *