How AI Search Finds and Selects Website Sources

When someone asks an AI search feature a question about your website, the feature must find relevant pages, assess what they say, and decide whether to use or cite them. Those steps depend on the search product. Google AI Search features use Google Search processes, while OpenAI documents a separate crawler for finding websites to show in ChatGPT search.

Access, indexing, selection, and citation are separate steps. A page can open in a browser but still be unknown to a search engine. A search engine can index it without finding it relevant to a question, and an AI feature can use it without citing it. Webmaster tools and site logs can help you check some steps, but no single check shows that an AI product will cite your page.

How a page gets into search

Search engines find web addresses, or URLs, through links and sitemaps, which list a site’s pages. They may then visit those addresses with automated programs called crawlers. After fetching a page, a search engine analyzes its content and may add information about it to an index, a searchable database. Google describes these stages as discovery, crawling, indexing, and serving results, and notes that not every page completes every stage in how Google Search works.

Each product has its own access rules. Google says its AI Overviews and AI Mode rely on the normal Google Search processes and eligibility requirements. OpenAI identifies OAI-SearchBot as the crawler used to surface websites in ChatGPT search; it documents GPTBot separately for potential model training. Allowing one crawler does not automatically allow another.

A crawler must load a page before it can assess its contents. Robots.txt rules, login requirements, server errors, or hosting and firewall settings can block access. For Google, a page also needs to return a successful response and contain indexable content to qualify for indexing. Meeting those conditions makes indexing possible, not guaranteed, according to Google’s technical requirements.

A search engine may fetch a page and still leave it out of its index. It can also choose a different URL as the canonical version, the one it treats as the main version among duplicate or similar pages. A sitemap or recrawl request can help Google discover a URL, but Google says neither guarantees indexing or a search appearance. Its recrawl guidance says a requested recrawl can take a few days to a few weeks.

How a search product selects sources

When someone submits a question, the search product finds pages that seem relevant and may pull information from several pages. Some AI features also expand the search. Google says AI Overviews and AI Mode may use “query fan-out,” meaning they run several related searches across subtopics and data sources. Google also says the two features may return different answers and links. That explains why a single question can involve more than one search, but Google does not publish a formula that predicts which pages will be chosen.

Microsoft describes Copilot as breaking content into smaller units that can be assessed for relevance and authority, then drawing on multiple sources to produce an answer. This is Microsoft’s description of its approach; it does not reveal a complete scoring formula or guarantee that a particular passage will be selected. The company’s AI search content guidance offers recommendations, but those recommendations do not establish a universal method for earning citations across products.

A crawler can access a page, a search engine can add it to its index, and a search feature can find it for a question. An answer generator can then use or cite it. A citation shows that the product presented a page as a source. It does not by itself show that the page supplied every fact in the answer, that the answer represented the page accurately, or that a user visited the site.

How to check your own site

Start with the pages that answer important customer questions. Open each page without logging in. Confirm that it returns a normal successful response, then check that its essential information appears as readable text. Review robots.txt and hosting or firewall settings for rules that could block the search crawlers relevant to your audience. These checks diagnose potential access problems; they do not prove a named crawler has visited the page.

For Google, use Search Console’s URL Inspection tool to check the reported indexing status and inspect the page as Google sees it. A site: search can help you find pages, but Google says it is not a definitive test of index status. If a URL is missing from results, use URL Inspection rather than treating the search operator as the final answer. Google reports traffic from its AI features within the overall Search Console Performance report under the “Web” search type, rather than as a complete record of every AI citation.

For Bing, use Bing Webmaster Tools’ URL Inspection tool to investigate crawling, indexing, and page issues. As of November 2026, Microsoft’s AI Performance report, announced in February 2026 as a public preview, reports citation activity across supported Microsoft AI experiences, including Copilot and AI-generated Bing summaries. It also shows cited pages and sample grounding queries, or phrases linked to the retrieval of cited material. Microsoft says these measures do not report rankings or a page’s importance within an answer. The AI Performance announcement describes the report’s scope and limits.

You can also review server logs for requests from identifiable crawlers, including OpenAI’s OAI-SearchBot. A logged request is evidence that a crawler requested a URL, not evidence that the search product indexed it or cited it later. Likewise, opening a URL successfully in your browser proves only that your browser can reach it.

Finally, test representative questions in the specific AI search products your audience uses. Record the date you ran the test, along with the question and product. Note any cited URLs, then repeat the check with realistic variations of the question. Treat the results as observations: Google says AI feature responses and links can differ, and one answer cannot establish stable visibility.

What to fix when a page is missing

If a page is blocked or unavailable, fix the access issue first. Check the crawler rules and hosting controls for the product you care about. For Google, do not use robots.txt to prevent indexing: Google explains that a crawler blocked from fetching a page cannot read a page-level noindex instruction. If you want Google to exclude a page from its index, Google’s documentation says to allow crawling and use noindex.

If search tools show that a page is accessible but missing from the index, check its status and canonical URL before you rewrite it. Make sure the intended page is linked from relevant pages on your site, and that a sitemap lists the preferred URL. A recrawl request can prompt Google to revisit a page, but it does not guarantee indexing or an AI citation.

If the page is indexed but does not appear for relevant questions, make its subject and evidence easier for readers to understand. State what the product, service, or page covers. Explain relevant limits and comparisons in visible text. Include facts there and support factual claims with evidence. These choices make the page clearer to people and provide searchable content; they do not guarantee that a particular search product will select it.

Structured data labels information on a page in a format search engines can interpret, so use it carefully. Google says its AI features require no special AI-only markup or schema beyond ordinary Search eligibility. Microsoft’s recommendations are broader, so do not assume that one provider’s advice guarantees a result on another provider’s service.

Where the checks stop

A successful fetch is not proof of indexing. Indexing is not proof that a page will match a specific question. A page that appears among retrieved sources is not guaranteed to appear in the final answer, and a citation does not measure visits, leads, or revenue.

Search Console reports Google Search performance in aggregate, while Bing’s AI Performance report covers supported Microsoft experiences and provides sampled grounding-query data. Neither provides a universal view of how every AI product discovers, selects, and uses website sources.

Key Takeaways

• Separate access from selection: A page can load successfully and still be absent from an index, unused for a question, or left out of the citations.
• Check the product you care about: Use Search Console for Google indexing, Bing Webmaster Tools for Bing diagnostics and supported citation reporting, and server logs for evidence of crawler requests.
• Fix access and clarity first: Allow the relevant crawler, link important pages internally, and put essential facts in readable text.
• Treat AI answer checks as samples: Record the product, question, date, and cited URLs, then repeat tests before drawing conclusions.
• Do not treat citations as business results: Citation reports and crawler logs do not establish traffic, conversions, or revenue.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com