What Sponsored Content Can and Cannot Prove About AI Recommendations
When an AI answer names a product or links to a page, the commercial context is not always obvious. “Sponsored” can describe…
On a website, an important page can lack a clear link from the rest of the site. A site crawler, a tool that follows links to find pages, may not reach it from its starting point. People often call such a page an orphan page. The term describes what the crawl found; it does not prove that no link to the page exists anywhere.
To find these pages, compare a crawl of your site with other URL lists. Then check whether each candidate deserves a place in your navigation. A sitemap or external link can make a page discoverable to crawlers while leaving it hard for visitors to find. A link audit should reveal weak paths to useful pages, not simply reduce the number of URLs flagged as orphaned.
A site crawler starts at one or more URLs and follows links it can recognize. If it cannot reach a page from those starting points, the crawl will not show that page as linked from another page on the site. The page might still appear in a sitemap, content-management-system export, Search Console report, analytics data, or server logs.
An orphan-page finding depends on the crawl’s starting points and settings. A page might have a link in a section the crawler missed, on a separate subdomain, or on another orphan page. Treat the report as a list of URLs to investigate, not a final diagnosis. Screaming Frog’s orphan-page method likewise compares a site crawl with other URL sources.
Also check whether existing links are useful. One vague link buried on an unrelated page might technically connect a URL to the site but offer little help to visitors. Google recommends using crawlable links from relevant pages and descriptive link text, while noting there is no ideal fixed number of links per page. Its link guidance explains how links help Google find pages and understand their context.
Start by identifying the pages you want people to find, such as current product or service pages, useful guides, and support resources. This gives you a way to prioritize findings. A raw count of every URL ever published does not tell you which pages need attention.
Next, crawl the site from its normal entry points, usually starting with the homepage. Before relying on the results, confirm which hostnames and sections the crawler includes. Check whether it follows links behind JavaScript. Review its crawl limits and exclusions too, since either could have stopped it from reaching a page. A crawl that omits a language section or subdomain cannot establish whether pages in that area have internal links.
Then compare the crawl with separate URL inventories. An XML sitemap can list URLs the crawler did not reach. Google says sitemaps can help it discover URLs, but inclusion does not guarantee that Google will crawl or index them; a sitemap also does not give visitors a useful navigation path. Check whether each sitemap entry is current and points to the preferred URL, as described in Google’s guidance to build and submit a sitemap.
Google Search Console, a tool for monitoring a site’s presence in Google Search, can show pages that appeared in search results (impressions) or received clicks. Its Performance report is useful evidence, but it is not a complete inventory of every page or visit. Google’s documentation on Search Console data explains that reports have coverage limits. An export from a content management system (CMS) can reveal published records missing from both the crawl and sitemap. Analytics and server logs can show URLs that received recorded visits or requests. Each source reflects what it records, so use several when available.
Give more attention to a URL when separate sources point to it. For example, if the sitemap lists a useful guide, the crawler cannot reach it, and Search Console reports impressions for it, inspect the page and its incoming links. The evidence still does not establish why Google found the page or whether the page should remain.
Compare the actual destination, not just the URL strings. A crawler and sitemap may represent the same page differently because of hostnames, trailing slashes, capitalization, query parameters, or redirects. Check where each URL resolves and which version the page identifies as its canonical URL, meaning the preferred address when equivalent versions exist.
Then decide whether the page should be available and discoverable. Confirm that it serves the intended content, is public, and still serves a purpose. If it is a duplicate, outdated campaign page, or old version of a better resource, linking it more prominently may create confusion rather than solve a problem.
For a page you want to keep, add a contextual link from a relevant hub, category page, or related article. Use link text that tells visitors what they will find. Then run the crawl again and confirm that it reaches the page through that path. Inspect the source page and the link itself; a larger incoming-link count does not show whether the path makes sense.
If the page overlaps with a stronger page, consolidate the useful material and redirect the old URL when appropriate. If it should no longer be available, remove it or use the appropriate setting to keep search engines from indexing it. Google distinguishes a noindex instruction from blocking a URL in robots.txt: a crawler must be able to access a page to read its noindex directive. See Google’s technical requirements before choosing how to handle a page you do not want in search.
A crawler only reports what it could reach under its settings. JavaScript rendering, robots rules, nofollow links, login requirements, URL exclusions, and crawl-depth limits can all affect the result. If a page appears orphaned, inspect a few representative pages and review the crawl settings before concluding that the page has no internal links.
URL inventories also have different blind spots. A sitemap can contain stale entries. Analytics covers only measured visits. Search Console reports Google Search data, so it does not cover all activity. A CMS export can include records that are not public. URL mismatches can also prevent a crawler from matching data from one source to another.
Finally, finding a URL or loading it successfully does not prove that Google indexed it, that it ranks, or that it produces business results. Search Console’s URL Inspection tool reports information about Google’s indexed view and offers a live test, but a successful live test does not guarantee inclusion in Google’s index. Use link audits to improve discovery and navigation, then measure search visibility separately.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com