Currently Available: Need a skilled Software Developer for your next project?
Categories
SEO, GEO and AEO

Building a Search Console API Integration for Internal Tools

For an internal tool that reports website search performance, a Search Console integration provides Google’s data on how a site appears in Search, including clicks, impressions, and average position. The Search Analytics API returns this data for a Search Console property, which identifies a domain or a specific URL prefix. Your code schedules requests, stores the results, and makes them available to internal users.

If your tool needs to choose when and how it imports data, schedule requests to the API. The integration needs access approved by a user, a reliable way to fetch each page of results, and storage that supports safe re-runs. The API limits and filters the rows it returns, so store how each result was requested and tell users that the export does not contain every search query.

Choose an export path

Use the Search Analytics API when your platform needs to set its own import schedule or request data for specific properties and date ranges on demand. Your code handles authorization, pagination, storage, retries, and monitoring.

Google’s managed BigQuery bulk export sends Search Console performance data to a BigQuery dataset daily, so you do not need to poll the API for it. It requires a BigQuery-enabled Cloud project and may incur storage and query costs. The export starts from setup and does not backfill earlier data. Google says it avoids the API’s daily row limit, but anonymized queries remain excluded. See Google’s bulk export overview and setup requirements.

The two paths can also work together: BigQuery can serve as the landing point, with a separate job loading data into an internal warehouse or tool. Choose based on where you want the ingestion responsibilities to sit. For a comparison of the available routes, see choosing between Search Console and BigQuery.

Authorize the user and select properties

The Search Console API uses OAuth 2.0 to let users approve access to their Google accounts. A read-only exporter should request the https://www.googleapis.com/auth/webmasters.readonly scope. An API key alone does not grant access to a user’s Search Console data, and the user must have permission for each property the integration reads. Google describes the Search Console authorization flow.

For scheduled requests from a server, the user approves access in a browser. The server exchanges the returned authorization code for an access token and, when issued, a refresh token. The access token authorizes API calls. The refresh token lets the server get a new access token after the first one expires. Store refresh tokens encrypted with access limited to the service that needs them; do not send them to browser code or write them to logs. Google’s OAuth documentation covers token use and storage considerations.

In a tool used by multiple people, link each credential to the account whose user approved it. Then let that user choose from the properties they can access. Use the Search Console Sites API to find properties if the interface needs to list them. Preserve the property identifier in the form Search Console uses, such as https://www.example.com/ for a URL-prefix property or sc-domain:example.com for a Domain property.

Schedule bounded requests

A Search Analytics request specifies a property and a date range with both start and end dates. It also specifies a search type, such as web, and optional dimensions, such as date, page, or device. Dimensions determine what one returned row represents. For example, a request grouped by date and page returns a row for each date-and-page combination represented in the results.

Google recommends querying one day at a time. Performance data is typically available after two to three days, while newer data can be incomplete. Schedule a pull for dates old enough to have settled, then revisit a configurable recent-date window to capture later updates. Google does not prescribe a universal length for that window, so make it a setting rather than a hard-coded promise. The requested dates use Pacific Time, which may differ from the internal platform’s reporting timezone. See Google’s guidance on performance data.

The following example assumes credentials already contains an authorized OAuth credential loaded from secure storage. It makes one request for a page of results:

from urllib.parse import quote

from google.auth.transport.requests import AuthorizedSession

def fetch_page(credentials, property_id, report_date, start_row=0):
    session = AuthorizedSession(credentials)
    encoded_property = quote(property_id, safe="")
    endpoint = (
        "https://www.googleapis.com/webmasters/v3/sites/"
        f"{encoded_property}/searchAnalytics/query"
    )

    request_body = {
        "startDate": report_date,
        "endDate": report_date,
        "type": "web",
        "dimensions": ["date", "page", "device"],
        "rowLimit": 25_000,
        "startRow": start_row,
    }

    response = session.post(endpoint, json=request_body, timeout=60)
    response.raise_for_status()
    return response.json().get("rows", [])

The Search Analytics API allows up to 25,000 rows in one response. To retrieve subsequent pages, increment startRow by 25,000 and repeat the same request until the API returns no rows:

def fetch_all_pages(credentials, property_id, report_date):
    rows = []
    start_row = 0
    page_size = 25_000

    while True:
        page = fetch_page(
            credentials,
            property_id,
            report_date,
            start_row=start_row,
        )
        if not page:
            break

        rows.extend(page)
        start_row += page_size

    return rows

Pagination retrieves successive response pages, but Search Console may still omit matching rows. Google documents a limit of 50,000 rows per day per search type and says the API may omit lower-priority rows. Detailed requests involving page or query dimensions can also lose data under internal processing limits. Record the query dimensions and explain these coverage limits to downstream users. The query reference describes request fields and response behavior.

Store results so re-runs are safe

Store the request details alongside the imported rows. At minimum, record the property, search type, and date. Also record the dimensions and filters used, when you retrieved the data, and the import status. Rows returned with different dimensions represent different groupings, so do not combine them as though they meant the same thing.

If you store rows in a consistent format, create a unique key, a value that identifies a row, from the property, search type, and dimension values in their requested order. First save all pages for a day in a temporary area. After every page has been fetched successfully, replace the stored rows for that property, search type, and date. This avoids duplicate rows on retries and prevents an incomplete run from appearing complete. If you keep raw responses or snapshots of each run, you can also compare imports when a later request returns different results.

A storage interface might look like this:

def ingest_day(credentials, property_id, report_date, store):
    run_id = store.start_run(
        property_id=property_id,
        report_date=report_date,
        search_type="web",
        dimensions=["date", "page", "device"],
    )

    try:
        rows = fetch_all_pages(credentials, property_id, report_date)

        store.replace_day_slice(
            run_id=run_id,
            property_id=property_id,
            report_date=report_date,
            search_type="web",
            dimensions=["date", "page", "device"],
            rows=rows,
        )
        store.mark_run_complete(run_id, row_count=len(rows))
    except Exception as exc:
        store.mark_run_failed(run_id, error=str(exc))
        raise

replace_day_slice represents an internal storage operation, not a Google API method. In production, make the write and run-status update transactional where your database supports it. A run should be marked complete only after all pages have been fetched and the stored slice has been updated.

Classify errors before retrying

A 400 response usually points to a request problem, such as an invalid date or dimension. The filter or property identifier may also be invalid, so correct the request rather than retrying it unchanged. A 401 or 403 can indicate an authorization that expired or was revoked. These responses can also point to missing property access or a quota issue. Inspect Google’s error reason and message: refreshing a token addresses expiration, while revoked consent or missing property permission requires the user to reconnect or restore access.

For 429, temporary server errors such as 500 or 503, and network failures, use exponential backoff with jitter. Increase the wait between retries up to a limit, and add a random delay so many scheduled jobs do not retry at the same moment. Save each attempt and its error response. If a run remains incomplete, alert the operator rather than treating partial data as a successful import.

Google documents separate request and load quotas for Search Analytics. Repeatedly querying the same data adds load, and staying below a request-rate limit does not guarantee that a costly query will avoid load-quota errors. Spread scheduled work across time and preserve the API’s error details in logs. The Search Console error reference and usage limits describe the documented errors and quotas.

Make reporting limits visible

Search Console’s API reports grouped performance data, not a complete log of every search. Anonymized queries are omitted from detailed query results even though they can contribute to totals shown in Search Console. Detailed rows therefore may not add up to the totals users see in the interface. Google explains this filtering in its performance data overview.

Label how current the data is. The API returns finalized data by default, while requests for fresh data can include periods that are still incomplete. If the integration requests fresh data, store that choice and any metadata about incomplete data with the run. Show the difference when reporting fresh and finalized results.

For issues involving pagination and missing rows, see handling export limits and pagination gaps. If API, interface, and export totals disagree, reconciling Search Console data can help explain why those views differ.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com

Email updates

Usually a new article and a few links I found interesting.

No spam. Unsubscribe with one click.

Leave a Reply

Your email address will not be published. Required fields are marked *