What AI Bot Logs Can Tell You About Access and Citations

If you run a website, logs from your server or content delivery network (CDN) show whether a request reached that layer, which URL it requested, and what response or security action the layer recorded. That evidence helps diagnose access problems, but it does not tell you whether an AI product indexed the page or cited it in an answer.

Each log records requests handled by the specific part of your site’s delivery setup that created it. A citation is a link an AI product shows as a source for an answer. These are separate facts: a crawler visit does not prove that an AI product cited the page, and checking citations does not show which request led to a citation. To investigate your own site, check the logs and access rules, then inspect citations in the products you care about.

What a bot request actually shows

An access log records requests seen by a server, content delivery network (CDN), firewall or another layer that handles website traffic. Depending on your logging setup, an entry may show the requested path and time, along with the reported user agent (the text that identifies the requester’s software). It may also include the source IP address, response status, and security action. Cloudflare’s HTTP request log documentation describes fields available in its logs; the fields you can see depend on your own provider and configuration.

A successful status code is evidence that the logging layer recorded a response. It does not establish that the crawler received the content you intended, understood it, indexed it or used it in an answer. A log from a CDN also describes what the CDN saw. It may not match the records at your origin server, especially when a security rule stops a request before it reaches the origin.

The bot’s specific name matters because services use different crawlers for different purposes. As of November 11, 2026, OpenAI documents OAI-SearchBot for search, GPTBot for potential training use, and ChatGPT-User for fetches prompted by a user. Those names describe different access purposes, and one bot’s visit does not establish activity by another. OpenAI’s crawler documentation explains the distinctions and the separate controls.

Google’s terminology works differently: Google-Extended is a robots.txt control token, not a separate user-agent string. Google says this control relates to certain Gemini uses of crawled content and does not affect inclusion in Google Search. Google’s crawler documentation explains the token’s scope. A search for a bot name in logs may miss some robots.txt preferences that apply to that service.

How to check your own logs

Start at the layer that handles your website traffic. If your traffic passes through a CDN, hosting provider, and origin server (the server that hosts your site), each may record different requests and security actions. A request blocked at the CDN can appear in edge logs and remain absent from origin logs. Check the log provider’s documentation to see which fields it records, whether it records only a sample of events, and whether it omits any events.

Then search for the specific crawler names published by the services you want to investigate. Review requests to page URLs as well as requests to /robots.txt: a visit to the robots file alone does not show that the crawler fetched an article. For each relevant entry, note its time and path, along with the reported user agent and source IP. Record the status too. Also note any block, challenge that asks the requester to pass a check, or rate limit that caps how often the requester can make requests. Compare those details with your CDN, firewall and hosting rules if the request failed.

Do not trust a user-agent string by itself. A request can claim to come from a known crawler without actually coming from that service. Google recommends checking crawler IP addresses against its published ranges or using reverse and forward DNS verification. OpenAI publishes IP information for its documented bots. Use the verification method each provider supports. If you cannot verify a request, treat it as unverified rather than confirming that a bot made the visit. See Google’s request verification steps.

Check robots.txt alongside the security rules. Robots.txt communicates crawl preferences to crawlers that follow its instructions; it does not enforce access restrictions. Google notes that a blocked URL can still appear in Search if other pages link to it. Use authentication or server-side restrictions for private content, and inspect edge rules for blocks or challenges that robots.txt cannot cause. Google’s robots.txt guidance explains what the file does and does not control.

A public check of vincentschmalbach.com on November 11, 2026, illustrates why a successful response needs a narrow interpretation. An ordinary, unauthenticated request to one article returned HTTP 200, and the selected article paragraphs appeared in the response HTML. The request did not impersonate a provider crawler. It confirmed what that particular request received at that time, not what any AI service fetched, indexed or cited.

Check citations separately

To find out whether an AI product cites your pages, test the product directly. Choose a small, stable set of relevant questions and record the date, exact wording, product and mode, whether the answer includes a source link, and the URL that link points to. Repeat the same questions later if you want to compare observations. Answers can vary, so one result is a snapshot, not a citation rate.

Check whether the cited URL belongs to your site or to a third party describing you. Your own logs only cover requests to the parts of your domain represented in those logs. They cannot tell you whether an AI product cited a review, directory, partner page or other site about your business.

Search tools provide different evidence from logs. For Google’s AI Overviews and AI Mode, Google says pages must be indexed and eligible to appear with a snippet to be eligible as supporting links. Eligibility does not guarantee that Google will crawl, index or show a page. Google’s guidance on AI features describes those conditions. A Google crawler entry is therefore useful access evidence, while Search Console and a direct citation check answer different questions.

Referral analytics add another view: they show visits that arrive through measurable links or referrers. They do not count every time an answer displays a citation, because a reader can see a citation without clicking it. A missing referral is not evidence that no citation appeared.

Where the evidence stops

A missing request in one log does not prove that a page was never fetched or cited. The log may be incomplete, sampled or limited to one layer. An AI service may use another index or fetch route, or rely on information it gathered earlier. The public documentation reviewed here does not provide a universal way to trace an AI answer citation back to a specific server-log entry.

A successful request also does not prove that a crawler processed the page as intended. Your site may redirect the URL or apply access rules. It may also return different content across requests. Any of these behaviors can change what a requester receives. If the response looks wrong, inspect the URL’s redirect behavior and the response at the layer where the log was recorded. Then compare the response with the relevant access rules. A test from your browser or terminal checks that request only; it does not reproduce every crawler’s environment.

Use logs to answer a specific question: which request reached which logging layer, and what did that layer record? Use direct product checks to answer whether a citation appeared. Keep those findings separate from claims about ranking, leads or business impact.

Key Takeaways

• Treat a bot visit as access evidence: A log can show a request and the recorded response or security action, but it cannot prove indexing or citation.
• Verify the bot’s identity and purpose: Check the service’s documented crawler names and verification method; a user-agent string alone can be faked.
• Inspect every relevant layer: Compare CDN, firewall, hosting and origin records, since a block at one layer may never appear in another.
• Test citations directly: Record the product, question, date and linked URL, then repeat comparable checks to see what changes.
• Keep outcomes separate: A citation, a click, a search impression and a business result are different measurements.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com