When Crawl Budget Is a Real Problem for a Website

For a website, crawl budget is the set of URLs Google is able and willing to crawl. Googlebot crawls a site by requesting pages and other resources from its server. Crawl budget becomes an SEO problem when Google repeatedly misses valuable pages or takes too long to revisit them, and evidence points to limited crawl capacity, low demand, or requests to unnecessary URLs.

A high crawl count alone does not show a problem, and a page missing from search results does not prove Google failed to crawl it. Google may have crawled the page and chosen not to index it. To diagnose crawl budget, identify the important pages, check whether Google reaches them, and look for evidence that server limits or low-value URLs are getting in the way.

How Google decides what to crawl

Google describes crawl budget through two related factors. Crawl capacity reflects how much Google can crawl without overloading a site. Google adjusts how much it crawls based on the site’s health and availability. Crawl demand reflects how much Google wants to crawl, based in part on the URLs it knows about. Their popularity matters, as does how often they change. Google does not publish a fixed crawl quota that applies to every website. Its crawl budget guidance focuses mainly on how those factors interact.

Google may crawl a site less even when the site has spare server capacity if Google sees little demand. If a site responds poorly, Google limits crawling even when it wants to fetch more pages.

The number and types of URLs Google finds also matter. Filters, sorting options, session identifiers, and other features can create many URL variations that Google can crawl. When Google spends requests on pages with little search value, it takes longer to find or revisit useful URLs. Google warns that extensive faceted navigation can create large numbers of URLs and use server resources.

When crawl budget deserves investigation

Google’s guidance gives rough scale examples for sites that may warrant closer attention: sites with at least 1 million unique pages that change about weekly, sites with at least 10,000 unique pages that change daily, or sites where a large share of URLs have the status “Discovered – currently not indexed.” These figures are guidance, not cutoffs. Google says smaller or less frequently updated sites generally do not need specialized crawl-budget work.

A site’s size alone does not show that it has a crawl-budget problem. Check whether Google has crawled important pages that can be indexed, and whether it revisits them in time for their content to matter. Then check whether the pages are hard to find, whether Google spends requests on unhelpful URL patterns, or whether server problems limit access.

The business or editorial need matters too. Delayed crawling is more consequential for frequently changing inventory or time-sensitive pages than for stable pages that rarely change. The goal is to determine whether Google is missing pages that matter, rather than to increase request counts for their own sake.

Check whether Google is missing the pages that matter

First, identify useful pages that should appear in search. Confirm that each is intended for indexing and accessible to Google. Compare those pages with the URLs Google can find through your site’s links and features such as filters. A large URL count can include many variations that do not need to appear separately in search.

Next, separate pages Google has not crawled from pages it crawled but did not index. In Search Console’s Page indexing report, “Discovered – currently not indexed” means Google has found a URL and has yet to crawl it. A large number of important URLs in this state merits investigation, but the status alone does not establish that crawl budget is the cause. “Crawled – currently not indexed” means Google fetched the page without adding it to the index. For those pages, investigate indexing and duplication issues instead of assuming that more crawling will solve the problem.

Use the Crawl Stats report to check trends in Google’s requests, response times, response codes, and host availability. Look for availability warnings or changes that coincide with server problems. The report covers most requests, but it is not a complete URL-by-URL history. Google also notes that sites with fewer than 1,000 pages generally do not need this level of crawl detail.

To see which URL patterns Google requests, inspect your server logs. Group Googlebot requests by page type, such as product pages, articles, redirects, and URLs with parameters. Compare those requests with the important pages you expected Google to reach. Google directs site owners to server logs for URL-level crawl checks, and notes that Search Console totals may differ from log counts. Logs show requests received by your server; they do not reveal every internal decision Google made about crawling.

A sitemap helps Google discover URLs, but it does not guarantee crawling or indexing. Keep it accurate and make important pages easy to reach through internal links. For more information, see Google’s sitemap documentation.

Fix the cause instead of chasing request volume

If Google spends requests on URL variations with little search value, reduce the number of unnecessary URLs the site exposes. For example, decide which filter combinations deserve search visibility and prevent unhelpful combinations from generating endless crawlable addresses. Do not assume that removing a URL pattern will transfer those requests to other pages. Google says it will not shift crawl activity unless it is already reaching the site’s serving limits.

If logs and Crawl Stats point to server trouble, fix the availability or response problem they show. Check the affected dates and URL groups, then confirm that Googlebot can fetch the important pages. Better availability helps Google crawl the site without overloading it, but does not guarantee that Google will crawl more.

Choose crawl and indexing controls according to the result you want. A robots.txt rule tells a crawler which URLs it should not request. A noindex directive tells Google not to include a page in search results, but Google must crawl the page to see that directive. Google’s guidance on blocking indexing explains this distinction. Blocking URLs to “free up” crawl budget does not reliably prompt Google to crawl other pages.

Canonical signals and sitemaps also do not compel Google to use a particular URL. A canonical is a signal about which version of a page you prefer; Google may select a different version. Before changing rules, check the specific URL’s response, crawl access, indexing directives, internal links, and canonical signals.

What crawl budget cannot fix

Crawling, indexing, and ranking are separate stages. Google first discovers a URL, then may crawl it, decide whether to index it, and later decide whether and where it should appear for a search. A page can fail at any of these stages for different reasons. Google’s troubleshooting guidance distinguishes crawling problems from indexing problems.

That distinction limits what a crawl-budget change can accomplish. Fixing a block or server error can make a page accessible to Google, but it does not guarantee indexing or better rankings. If Google already crawled a page and chose not to index it, focus on the page’s indexing and duplication issues rather than trying to raise crawl volume.

What this means for Google AI search

Google says its AI Overviews and AI Mode use the same general SEO foundations as other Search features. A page must be indexed and eligible to appear in Google Search with a snippet to be eligible as a supporting link, but meeting those requirements does not guarantee that Google will crawl, index, or show it. Crawl access is therefore a prerequisite for eligibility in these features, not evidence that crawl-budget work will produce an AI citation. Google’s AI features guidance describes those limits.

Key Takeaways

• Look for missed important pages: Crawl budget deserves attention when valuable, indexable URLs remain uncrawled or are revisited too late, and the evidence points to a crawl or server constraint.
• Separate crawl status from index status: “Discovered – currently not indexed” means Google has not crawled the URL; “Crawled – currently not indexed” means it has.
• Compare reports with server logs: Use Crawl Stats for site-level trends and logs to see which URL patterns Googlebot requested.
• Remove the actual bottleneck: Improve access to important pages and reduce low-value URL variations, but do not expect blocked URLs or sitemap entries to reallocate or guarantee crawling.
• Keep AI-search expectations narrow: Crawl access supports eligibility in Google Search features, including AI features, but does not guarantee inclusion or citations.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com