Validating Search Console Bulk Export Partitions and Retries
When Search Console sends performance data to BigQuery, it writes each day's data into date-based partitions, which keep each date's rows together.…
When Google appears to crawl a site less often, Search Console’s Crawl Stats report helps determine whether Google encountered trouble reaching or fetching the site. The report summarizes Google’s requests to crawl the site, the responses servers send, and whether Google can reach the site’s host, or web server. Crawl demand means how much Google wants to crawl; Crawl Stats does not directly measure it.
Look for access problems by checking whether host-availability issues and failed responses occurred on the same dates. If availability and responses look healthy, low crawl volume alone does not show that the site is blocking Google. You will need to check whether Google has found the relevant URLs and whether it has a reason to crawl them more often.
Examine dates and response patterns before considering the total number of requests. Crawl Stats summarizes request counts and response categories. It also reports file types, download size, average response time, and host availability. You can select report entries to inspect example requests. Google describes the report in its Crawl Stats documentation.
Host availability tracks three areas: whether Google can fetch the site’s robots.txt file, which gives crawlers rules for URLs; resolve the site’s hostname through DNS, which maps hostnames to network addresses; and connect to the server. The report summarizes significant issues across the previous 90 days and distinguishes recent issues from those that occurred more than a week ago. Open the relevant graphs and check the dates and details rather than relying only on the summary status. Google gives DNS failures above 5% of requests in a day as an example of a threshold breach for that category; it is not a universal threshold for every availability chart.
A cluster of 5xx responses, which are HTTP status codes for server errors, timeouts when Google does not receive a response in time, DNS failures, or connection errors around an availability issue points to a need to investigate access or serving capacity. Compare those dates with the example requests and, where available, hosting, DNS, firewall, and content delivery network logs. Rising average response time adds context, particularly when errors rise too, but it is an aggregate across fetched resources and does not measure page-rendering speed.
Successful responses need context as well. A 200 response means the server returned content; expected 301 or 302 redirects and 304 responses can also be normal. A 404 is not automatically an access failure if the URL was intentionally removed. Check whether the URL should still exist and whether its response matches that expectation.
For example, suppose requests fall during a period when 5xx responses and timeouts rise, and the host-availability report shows a recent issue. Investigate hosting capacity or an intermediary such as a firewall when you see that pattern. If requests fall while responses and availability remain healthy, the report alone cannot tell you whether Google has low demand, has not discovered important URLs, or is encountering a separate crawl restriction.
A robots.txt rule and a robots.txt fetch failure describe different problems. A rule in the site’s robots.txt file tells Google not to crawl matching URLs. That is a crawl restriction; it does not mean Google tried and failed to fetch each blocked page. To check whether a particular URL is blocked, review the robots.txt rules and use URL Inspection or relevant details in the Page Indexing report.
A robots.txt fetch failure means Google could not reliably retrieve the file it checks before crawling. That is an access issue, and Google’s Crawl Stats guidance says crawling can slow or stop while Google cannot get an acceptable response. A 404 response for robots.txt means Google found no such file; it is treated as a successful response, rather than as an unavailable file.
Other failure categories point toward different checks. A 5xx response means the server returned an error; a timeout means Google did not receive a response in time. DNS errors indicate a problem resolving the hostname. A redirect error can involve too many redirects or a loop. A 401 or 407 response indicates an authorization requirement, so check whether the resource is intended to be public and crawlable. These categories narrow the investigation, but they do not always identify the underlying cause.
A blocked URL can still be known to Google even though Google has not fetched its contents. Robots.txt controls crawling; it does not reliably keep a URL out of search results. Google’s robots.txt documentation explains this limitation. If the goal is to prevent access or indexing, choose a method suited to that goal, such as password protection for private content.
Use the report as one part of a diagnosis:
These checks help you distinguish a sitewide access pattern from an issue affecting one URL. However, Search Console does not provide a crawl history filtered by URL or path. For the history of a particular URL, use server logs where available; URL Inspection tests a URL but does not act as a historical crawl log. Google outlines these checks in its crawling-error troubleshooting guidance.
Google describes crawl budget as a combination of crawl capacity and crawl demand. Capacity concerns how much crawling Google can send without overloading the site. Slow responses, 5xx errors, and rate limiting can reduce it. A 429 response tells a client that it has sent too many requests. Demand concerns how much Google wants to crawl, based on factors such as the site’s size, update frequency, and the perceived importance of its URLs. Google explains the distinction in its crawl-budget guidance.
Healthy availability removes one possible access constraint. It does not show that Google has discovered every page, wants to crawl more often, or has indexed the URLs you care about. Likewise, a lower request count does not by itself establish a server problem or a crawl-budget shortage. Check URL discovery, robots rules, the sitemap, and Page Indexing details before changing server capacity in response to a low crawl total.
Crawl Stats also counts requests, not unique pages. Repeat requests for one URL count separately. Requests for resources on the property also count. Crawl Stats counts each step in a server-side redirect chain as well. Its response percentages show the share of request events in each response category; they do not show what percentage of distinct URLs are broken.
Crawl Stats does not provide a direct score for crawl demand or a number of URLs Google still wants to crawl. A low request count cannot distinguish low demand from undiscovered URLs or crawl restrictions. A healthy host status means the report did not record significant availability issues in its 90-day window; it does not prove every URL, resource, or access route worked at all times.
The report and server logs also do not have to match exactly. Google says some crawl requests may not appear in the report. During robots.txt problems, the total can include potential fetches Google considered but did not make. And a request that never reached the server, such as one affected by a connectivity problem, will not appear in that server’s logs. Treat both sources as complementary evidence, and use their dates and examples to investigate rather than expecting identical totals.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com