How to Test Redirects for a Specific Search Crawler

When a URL redirects, send a request to the starting URL, follow each redirect, and request the destination directly to check whether a particular search crawler can reach it. A redirect is an HTTP response that tells a client to request another URL. A crawler is an automated client that fetches web pages for a search or other service.

“AI crawler” does not identify one bot. As of November 11, 2026, OpenAI documents OAI-SearchBot for ChatGPT search, GPTBot for possible model-training collection, and ChatGPT-User for certain user-initiated requests. Each name refers to a different purpose. Identify the crawler or product you want to test, because a test of one does not show how another behaves.

Check the redirect response and destination

First, send a normal GET request to the exact URL that should redirect. Use the right scheme and hostname for the URL. Include the intended path and query string. A GET request asks the server for the page. The curl -I option sends a HEAD request instead, so do not rely on it alone.

curl -sS -D - -o /dev/null 'https://example.com/old-path'

Look for a 3xx status and a Location header, which names the next URL. The HTTP standard defines 301 and 308 as permanent redirects and 302 and 307 as temporary redirects. For a normal page request using GET, the method distinction between these codes is usually less relevant than for a form submission or another request type.

Then follow the redirect chain and record where it ends:

curl -sS -L -D - -o /dev/null --max-redirs 10 \
  -w '\nfinal_url=%{url_effective} status=%{http_code} redirects=%{num_redirects}\n' \
  'https://example.com/old-path'

The output shows the response headers for each step, the final URL and status, and how many redirects curl followed. Curl’s -L option follows redirects. This test shows what curl fetched from the machine and network where you ran it. It does not reproduce everything a search crawler does.

Request the destination directly as well:

curl -sS -D - -o /dev/null 'https://example.com/new-path'

If the redirect crosses to another hostname, check that destination host too. A correct first response does not help if the destination returns an error, requires a session, or is blocked by a firewall.

Test the crawler-specific rules

After checking the ordinary response, compare it with a request that uses the crawler’s documented user-agent string. A user agent is text in an HTTP request that identifies the claimed client. Curl’s -A option sets that text; use the exact current string from the provider’s documentation.

To try this, add -A '...' to the curl command and replace the quoted placeholder with the documented string. The response can reveal rules on your server, CDN, or firewall that change based on the user agent. It does not turn curl into that provider’s crawler. Anyone can copy a user-agent string, and the real crawler’s IP address, request headers, location, or security checks may produce a different response.

Check the robots.txt file, which lists crawl rules, and other access controls for both the starting URL and destination. For OpenAI, check the product token separately: allowing OAI-SearchBot does not automatically allow GPTBot. OpenAI’s crawler documentation, checked November 11, 2026, describes its crawler roles and IP ranges but does not specify a redirect-hop limit or provide a dedicated redirect test. Do not infer those details from a user-agent simulation.

Confirm what the provider’s tools and logs show

A command-line request answers what your client received. To check whether a real crawler requested the URL, inspect logs from the origin server (the website’s server), CDN, or firewall for the starting URL and each destination. Look at the timestamp, requested path, response status, Location value, source IP, and user agent. Verify the source IP using the provider’s instructions; a matching user-agent string alone does not prove the request came from that crawler.

Google and Bing provide crawler-verification guidance. Google’s Googlebot documentation explains how to check reverse and forward DNS or compare an address with its published crawler ranges. Bing offers a Bingbot verification tool. OpenAI’s crawler documentation lists IP ranges for its crawlers.

Provider inspection tools also have limits. Google says its general crawlers follow redirects, but Google’s Inspection Tools do not. Bing Webmaster Tools’ Live URL test also stops at a redirect and directs you to inspect the destination separately. These tools test their own inspection requests, not every step of routine crawling. Read the provider’s current instructions before treating an inspection result as evidence about a particular redirect path.

Diagnose loops, chains, and different responses

A redirect loop occurs when URLs send requests back to one another or repeatedly change the scheme, hostname, or path. Curl’s followed-response output helps expose the sequence. A missing or incorrect Location header points to a problem in the first redirect response. If the destination returns 403, 404, or 5xx, investigate that response separately; the first URL may be redirecting correctly while the target remains unavailable.

Each redirect in a long chain adds a request and makes failures harder to trace. Google says its general crawlers follow up to 10 redirect hops, while recommending that site owners send users and crawlers directly to the final destination where possible. For a chain that cannot be removed, Google recommends keeping it short, ideally no more than three redirects and fewer than five. Those figures describe Google’s guidance. Other crawlers and tools may have different limits. See Google’s documentation on HTTP status codes and crawling.

A server can make a browser appear to redirect by returning a 200 page that uses JavaScript or a meta refresh to send it elsewhere; an HTTP redirect returns a 3xx response with a Location header. Google supports several ways to redirect but recommends redirects handled by the server where possible in its redirect guidance. If curl sees a 200 and the browser changes URL, check the page’s HTML and scripts to identify the client-side behavior.

Finally, compare responses for the same URL across normal and crawler-like requests. If they differ, inspect CDN caching, bot-management rules, geographic restrictions, authentication, and firewall challenges. Treat the difference as a lead for investigation, not proof of what the genuine crawler receives, until verified logs or a provider tool support that conclusion.

What a successful test proves

If curl receives the expected redirect and fetches the destination, the route works from that client and network. A verified crawler request in logs provides evidence that the crawler requested the recorded URL and received the recorded response.

Neither result proves that the crawler indexed the destination, selected it as canonical, showed it in search, cited it in an AI answer, or affected rankings or business results. Those outcomes involve systems beyond the redirect exchange. Report the crawler or tool, the URL tested, and the observed responses so readers can see exactly what the evidence covers.

Key Takeaways

• Name the crawler first: Search, training, and user-triggered fetchers have different purposes; testing one does not test the others.
• Inspect both ends: Check the initial 3xx response, every redirect hop, and the final destination with GET requests.
• Treat user-agent tests as simulations: A copied user-agent string can expose your own rules, but it does not authenticate a real crawler.
• Verify real visits: Use server or CDN logs and the provider’s IP-checking guidance to confirm crawler requests.
• Limit the conclusion: A successful fetch shows access from a specific client or verified request, not indexing, citations, rankings, or business impact.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com