How to Verify AI Crawler Traffic Instead of Trusting Its User Agent

When an AI crawler visits your website, its request usually includes a user-agent string, text that names the software the requester claims to use. A request labeled GPTBot, ClaudeBot, or PerplexityBot helps you find possible crawler visits in your logs, but the label does not prove who sent it. Any requester can use another crawler’s name.

To verify AI bot traffic, compare the IP address the request came from with the crawler operator’s current official records. If a content delivery network (CDN) or proxy sits in front of your site, first make sure your logs capture the visitor’s IP address rather than the proxy’s. A verified request shows that the operator lists the request’s IP address for that crawler. It does not show that a page was indexed, cited, or used in an AI answer.

Find the real source IP

Start with access logs from the service that first receives public traffic: often a CDN, firewall, load balancer, or web server. Find requests whose user-agent claims to be an AI crawler, then note the timestamp, requested URL, response status, user-agent, and source IP. The user-agent helps you locate candidates; treat it as a claim until you check the network source.

If a CDN or reverse proxy, a server that forwards requests to your origin, handles incoming requests, your origin server may record the proxy’s address instead of the crawler’s. Cloudflare documents how to restore and log the original visitor IP for proxied requests. Use the equivalent instructions for your provider, and trust a forwarded-IP header only when a trusted proxy sets it. If requests can reach your origin directly, a requester could supply a forged header.

Check the IP address where the incoming connection first arrives, or confirm that a trusted, correctly configured proxy passes the client’s IP address to your origin logs. If you cannot establish which address belongs to the requester, mark the request as unverified rather than comparing the proxy’s address with a crawler list.

Match the IP to the operator’s current records

Use the crawler operator’s own documentation, not a copied list from a bot directory. Operators publish different identity information, and some separate their address lists by crawler purpose.

As checked on November 11, 2026, OpenAI’s crawler documentation lists separate user agents and IP information for crawlers with different roles. Check the list for the specific agent in the request. A match for one OpenAI crawler does not verify a request claiming to be another. Anthropic’s current crawler guidance points site owners to its source-IP information. Perplexity’s crawler documentation links to its published address list, which is also available as PerplexityBot’s JSON file.

Compare the logged IP address with the right list, including any network ranges. Some lists use CIDR notation, a way to describe a range of addresses instead of one address. Use a comparison that checks whether an IP address falls within a listed range and supports the address format in the operator’s list, including IPv6 when applicable. Record when you retrieved the list: operators can change their infrastructure, and a current list alone may not establish which addresses were valid when an older request occurred.

Google documents another way to verify its own crawlers using the Domain Name System (DNS), which connects IP addresses and hostnames. Reverse DNS looks up the hostname associated with an IP address. Forward DNS then checks whether that hostname points back to the original IP address. Google requires both checks for its documented procedure. That method is specific to Google’s published guidance; use it for an AI crawler only if that operator provides a comparable procedure. See Google’s verification steps.

Label what the check establishes

Record exactly what you verified for each request: whether its IP address matched an official list, it passed the operator’s documented DNS checks, only its user-agent matched, or you could not verify it. Record the crawler name, request time, URL, and response status. Also note the date of the identity list you checked.

Then decide access based on your site’s policy and the request’s behavior. A verified crawler may still consume resources your site does not want to provide, so you can choose whether to allow it. Treat a familiar user-agent as insufficient grounds for special access to an unverified request. You can allow, rate-limit, or block traffic according to your needs.

Keep crawler identity separate from crawler purpose. For example, OpenAI documents different agents for search, training-data collection, and user-triggered visits. A verified request from one agent does not establish that another agent visited, and a user-triggered fetch is not the same as automated crawling. Check the operator’s description of the specific agent before interpreting the request.

Where verification stops

An IP match supports the conclusion that a request came from an address the operator publishes for a particular crawler. It does not prove what the operator did with the page afterward. A server log records that a crawler visited and shows the response your server returned. It does not prove that the page was indexed, appeared in an AI answer, received a citation, changed rankings, or affected your business. Google likewise states that crawling does not guarantee indexing in its technical requirements.

A list can also be stale, a proxy can obscure the original IP, and a user-agent can be forged. If the operator does not publish usable identity evidence for the specific crawler, classify the request as unverified rather than treating the user-agent as proof.

Cryptographic signatures give some systems a stronger check by attaching data to a request that lets the receiver verify whether parts of it are authentic. The HTTP Message Signatures standard defines a way to sign and verify parts of an HTTP request. Cloudflare documents Web Bot Auth, which uses signatures and published public keys. This approach works only when the crawler signs requests and the receiving service verifies them correctly; it is not a universal feature of AI crawlers.

Key Takeaways

• Treat user-agents as claims: Use them to find candidate requests, then check the source IP against the operator’s official records.
• Verify the client IP first: If a CDN or proxy sits in front of your site, confirm that the logged address belongs to the requester and that trusted proxy settings protect forwarded-IP headers.
• Use the right operator list: Check the current official record for the specific crawler, since providers can separate address ranges by purpose and update them.
• Record uncertainty honestly: Label a request as verified, user-agent-only, mismatched, or unverified, and note when you checked the address list.
• Separate access from outcomes: A verified fetch shows that a crawler reached a URL; it does not prove indexing, citation, or business results.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com