How to Diagnose a CDN or Firewall Blocking Search Crawlers

If Google or another search engine cannot fetch a page, a content delivery network (CDN) or firewall might be denying the request. A CDN serves website traffic through a network of servers. A web application firewall (WAF) checks incoming requests against security rules. Either one can block or challenge a crawler before the request reaches your website’s server.

A page loading in your browser does not show what the crawler received. To diagnose a suspected block, match a request from a verified crawler to the response and the rule or server log that handled it. This also helps separate search-engine access from settings that target AI crawlers, such as Cloudflare’s AI-bot controls.

Find out which layer handled the request

A crawler’s request can pass through several layers: a CDN, a WAF, a hosting-provider firewall, the website’s server, and the site itself. A rule at any of these layers can block the request, slow it down, or return a challenge instead of the page.

The response offers clues, but a status code alone does not identify the cause. A 403 means the request was denied; a 429 indicates rate limiting; and a 503 reports that the service could not handle the request. Each could come from different parts of the setup. A challenge page can also come with a status code that looks successful. Cloudflare describes challenges as checks that can interrupt a request before the visitor reaches the website.

Start with one affected URL and a time when access failed. Compare the CDN or WAF event records with the access and error logs from the origin server, the server that hosts your site. Check the same URL and time in both. Look for the action taken, the rule or service responsible, and the response returned. If the CDN records a request but the origin does not, the CDN may have served a cached copy or handled the request itself. That gap is a clue, not proof of a block: logs may be incomplete, or another layer may have stopped the request.

Verify the crawler before changing a rule

A request can use Googlebot’s or another crawler’s name in its user-agent string, but that easily copied string does not prove who sent it. Before allowing a crawler through a security rule, follow the operator’s verification instructions. Google documents how to verify requests from its crawlers.

Use your CDN’s event details and request logs to check the request’s source information according to the crawler operator’s verification instructions. If you cannot verify the source, treat the user-agent as a claim. Allowing every request that uses a trusted crawler’s name could also let impersonating traffic through.

Test tools help reveal how your own request is handled, but they do not establish how a verified crawler was treated. A curl request with a Googlebot user-agent still comes from your test machine, not Google. A browser may pass a challenge that a crawler cannot. Cloudflare’s documentation explains that some challenges depend on browser capabilities, so a command-line test and an actual crawler request can receive different results.

Check the rule and the crawler category

Search crawlers and AI crawlers are separate categories of automated traffic. Tools that fetch pages at a user’s request are another category. A setting intended to block AI crawlers does not, by itself, show that Googlebot or Bingbot is blocked. First identify the crawler involved; then find the rule or security feature that acted on its request.

As of November 11, 2026, Cloudflare documents separate controls for managing AI crawlers and for custom bot-protection rules. Other WAF rules and bot protections can affect requests. Rate limits and path-based conditions can affect them too. Check the actual event and rule action instead of relying on a dashboard setting’s name.

Cloudflare’s Security Events records can show which security feature acted on a request. Available records depend on the plan and data view, and some views use sampled events. An empty search therefore does not prove that no block occurred. Also check any separate hosting firewall, server rule, security plugin, or proxy. Cloudflare’s records cannot show what a separate layer did.

Match the evidence to the conclusion

The strongest evidence of a block is a verified crawler request that matches a WAF or firewall event showing that a rule challenged or denied it. That finding confirms what happened to that request. It does not establish that every URL or crawler receives the same treatment.

Other observations support narrower conclusions:

A robots.txt file tells crawlers which URLs to fetch. It does not control access. Google’s robots.txt documentation explains that the file is not a security measure. If you need to keep content private, use authentication or another access-control method. If your aim is to prevent indexing, Google advises allowing the crawler to fetch a page carrying a noindex directive; blocking that page in robots.txt can prevent the crawler from seeing the directive.

Adjust the rule and confirm the result

If the logs show that a rule blocked the request, change that rule. Keep the change as narrow as the evidence allows, such as adjusting a condition for a specific path, rather than weakening protection across the site. Use the provider’s documented verified-bot signal where available. Do not create a broad exception based only on a user-agent string.

If a challenge caused the failure, check whether the crawler can complete that challenge and whether the rule needs to challenge that traffic. Then retest the affected URL and inspect fresh events and server logs. For Google, use Search Console’s URL Inspection and crawl data to investigate Google-specific access. Those checks do not diagnose how other crawlers or fetchers are handled.

A successful retest confirms that the test worked under those conditions. It does not prove that Google will index the page or that any search engine will rank it. Google states that meeting its technical requirements does not guarantee inclusion in its index. Crawling, indexing, ranking, and use in AI-generated answers are separate outcomes.

Key Takeaways

• Match the request to the rule: Use the URL, time, response, and security event to identify which layer acted.
• Verify crawler identity: A Googlebot or Bingbot user-agent string alone does not prove the request came from that crawler.
• Separate AI and search access: A Cloudflare AI-crawler setting does not by itself confirm whether a search crawler is blocked.
• Treat tests as limited evidence: A successful browser or command-line request shows what that test received, not what a verified crawler received.
• Make narrow changes: Adjust the rule shown in the logs, then check the affected URL and new records while preserving unrelated protections.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com