How to Use Search Console Crawl Stats to Investigate Access Problems

When Google has trouble reaching pages on your site, Search Console’s Crawl Stats report can show whether Google recorded problems with your server connection, DNS, or robots.txt. DNS is the service that directs a hostname, such as example.com, to the server that handles its requests. robots.txt is a file that tells crawlers which parts of a site they may access.

Use Crawl Stats to find a pattern and when it occurred, then check server, hosting, content delivery network (CDN), firewall, and DNS records to identify the cause. The report summarizes Google’s crawl activity; it does not provide a complete request log or prove that a crawled page was indexed.

Start with Host status

In Search Console, open Settings → Crawl stats for a property that covers a host’s main address. Google provides the report for Domain properties and URL-prefix properties that cover a host’s root address, rather than just a specific page. If your property includes several hosts, check which host the data represents before investigating a specific subdomain.

Start with Host status, which reports Google’s ability to fetch robots.txt, resolve the host through DNS, and connect to the server. Compare the graphs with the dates of a suspected outage, DNS change, firewall update, or deployment. The report’s host-status information covers the previous 90 days. A green status means Google found no significant availability issue in that period; it does not guarantee that every URL remained reachable at every moment. Google’s Crawl Stats documentation explains the report’s status categories and scope.

If a graph shows an availability problem, open that category and note the dates and example URLs. Google provides a red threshold line to flag possible issues. For example, the documentation describes DNS failures on more than 5% of requests on a given day as a threshold breach. Treat that signal as a reason to investigate the relevant infrastructure, not as a diagnosis of the cause.

Match the error pattern to the infrastructure

Next, inspect the breakdown of HTTP response codes, which show how servers answered Google’s requests. Look for a change that coincides with the Host status issue, rather than judging the site by the total number of crawl requests alone.

Google’s crawling-error troubleshooting guidance recommends correlating crawl data with site issues and checking infrastructure such as firewalls when requests time out or connections fail. A “Page could not be reached” result is especially relevant: Google says that request did not reach the server, so it will not appear in the server’s access logs. In that case, check DNS and firewall records, along with CDN and network records. Also check origin-server logs.

Handle an unavailable robots.txt differently from an ordinary 404: Google treats that as a successful response when the file returns 404, while a connection problem that prevents Google from fetching it can affect crawling. Check the actual response and delivery path before changing crawl rules.

Use the other graphs to narrow the problem

Crawl Stats also groups activity by file type, type of Googlebot (Google’s web crawler), and crawl purpose. These views can help show whether an incident appears to affect a particular kind of resource or crawler. For example, a concentration of failures for JavaScript or image files points to a different set of URLs to inspect than a broad rise in HTML server errors. A shift between Discovery requests for URLs Google has not crawled before and Refresh requests for known URLs provides context, but does not establish an access failure by itself.

Click example URLs to find leads for further checks. Google describes these as representative examples, not a complete list. The report counts requests, so repeated requests for one URL count more than once. Its request totals and response percentages therefore do not describe the number of unique pages affected.

Confirm the problem with URL Inspection and logs

For a specific page, use URL Inspection, a Search Console tool for checking how Google accessed a URL, to see whether Google reports an access or host-load problem. Then compare the reported time and URL with your server and infrastructure records. Search Console does not provide a crawl-history report that you can filter by URL or path, so logs are the better source for detailed request history. Google’s troubleshooting documentation also notes that requests that never reach your server will not appear in its logs.

Search Console and server-log totals may differ. Crawl Stats can include potential fetches that Google abandoned because robots.txt was unavailable, and Google says that some crawl requests might not be counted in the report. Compare equivalent time periods and remember that the two sources measure activity differently.

Check indexing separately if the page is missing from search. Crawl Stats helps diagnose Google’s access to the site; it does not show whether Google indexed a page. Use the Page Indexing report and URL Inspection to investigate indexing status. Google’s technical requirements describe the distinction between allowing Google to access a page and having that page appear in Search.

Read crawl volume as a clue, not a verdict

A drop in total crawl requests alone does not prove that Google is blocked. Google adjusts crawling in response to both its capacity to crawl a site and its demand for the site’s URLs. Demand depends on factors such as site size, how often it changes, its popularity, and how valuable Google considers its content. Google’s crawl-budget guidance explains why crawl volume can change without a new access failure.

A stronger case for an access problem combines several signals: a recent Host status issue, a rise in a matching error category, affected example URLs, and infrastructure records that line up with the same period. A lower request total, a single example URL, or a change in average response time alone is weaker evidence. Fix a confirmed availability problem, but do not assume that more crawl requests will automatically produce more indexing or higher rankings.

The same distinction matters for Google’s AI features in Search. Google says a page must be indexed and eligible to appear in Search with a snippet before it can serve as a supporting link in AI Overviews or AI Mode. Crawl Stats can help identify access problems that interfere with crawling, but it does not report whether a page appears in those features. Google’s guidance on AI features describes their eligibility requirements.

Key Takeaways

• Check Host status first: Compare robots.txt, DNS, and server-connectivity graphs with the dates of suspected incidents.
• Match errors to infrastructure: Use the response breakdown to decide whether to inspect DNS, access rules, redirects, or server and network records.
• Treat example URLs as leads: Crawl Stats shows patterns and representative requests, not a complete URL-level history.
• Verify the cause outside Search Console: Compare the report with records from the server, CDN, firewall, hosting, and DNS. Some failed requests never reach the origin server.
• Separate crawling from indexing: Use Page Indexing and URL Inspection to check whether Google indexed a page after it could crawl it.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com