Currently Available: Need a skilled Software Developer for your next project?

SEO, GEO and AEO

A Practical SEO and GEO Implementation Guide

November 11, 2026 by Vincent Schmalbach

Start with business questions

Before choosing keywords, URLs, or technical changes, decide which questions the website must answer. Start with the products, services, locations, or software the business provides. Then identify the people who choose, buy, use, or manage them and the information each person needs to act.

Collect real questions from support requests, sales conversations, site search, product documentation, and customer feedback. Group them by the reader’s task: choosing a product, using it, comparing options, understanding pricing, or fixing a problem. People phrase the same concern in different ways. Organize content around the decision behind the question.

Tie each answer to evidence

For every question, record its audience and the decision it supports. Then note the accurate answer, its supporting evidence, the page that should answer it, and the reader’s next useful action. Evidence includes current product documentation and published terms. It can also come from verified specifications or confirmation from the person responsible for the service. Record any conditions that change the answer. These may include plan level, location, product version, or effective date.

For example, a software provider may hear repeatedly, “Does this plan include single sign-on?” The relevant page should give the current answer, link to plan details or product documentation, and state any prerequisites. The next action may be comparing plans or configuring the feature. If product documentation and a sales page conflict, correct the conflict before publishing more copy.

This record separates a missing answer from a missing page. An existing product page or support article may already be the right place to answer the question. Create a separate page when the question represents a distinct reader task. Do not create near-duplicate pages for small differences in search wording.

Measure the business result

Define the business outcome separately from discovery measures. A product page may seek qualified sales inquiries. A help page may seek successful setup by an existing customer. Search impressions, visits, and citations in an AI assistant response measure exposure or discovery. They do not, by themselves, show that a page produced an inquiry, sale, or completed setup.

Google’s AI features documentation, updated December 10, 2025, states that a page must be indexed and eligible to appear in Google Search with a snippet before it is eligible to appear as a supporting link in AI Overviews or AI Mode. Google specifies no additional technical requirements for these features. This guidance covers Google Search products only. It does not describe how other AI assistants select sources, and eligibility does not guarantee inclusion in an answer.

A starting record needs seven fields: question, audience, verified answer, supporting evidence, page, reader’s next action, and business outcome. Use it to assign each answer to an existing or planned page and identify who keeps the facts current.

Inventory pages and owners

Turn the questions into a URL inventory: one working list that connects every page to its reader, purpose, technical state, and accountable owner. A site crawl finds URLs through links it can follow. It misses URLs that appear only in the CMS, analytics reports, Google Search Console, XML sitemaps, product catalogs, or help-center platforms. Export URLs from each source, note the source, and reconcile duplicates.

No single export shows the whole site. An XML sitemap helps search engines discover URLs, but inclusion does not establish that a page should be crawled, indexed, or retained. Google describes sitemaps as a discovery aid rather than a guarantee of crawling or indexing in its sitemap guidance.

Keep one row for each URL

Use a spreadsheet or database with one row for each distinct URL. Record the page’s purpose, intended audience, content owner, page type, language, and current status. Also note its intended indexability, canonical URL, sitemap membership, and important internal links. Indexability means whether the page is intended to appear in search results. Keep the discovery source, such as “CMS,” “support export,” or “site crawl,” so reviewers know why the URL appears in the inventory.

For example, a SaaS help page may be public English-language documentation, owned by a support-content editor, and intended for indexing. A billing screen may be login-restricted transactional content that should remain out of search results. Those facts prevent an audit from treating both URLs as ordinary public pages.

Group URLs by function: home and navigation, product or service pages, documentation, editorial content, support answers, transactional pages, and utility or private pages. These groups reveal template-level faults while preserving differences in page purpose and reader needs.

When reconciling URLs, merge obvious variants, such as alternate hostnames or trailing-slash versions, only after confirming that they serve the same content. Preserve query parameters when they change the page content or reader task. Keep language versions as separate entries even when they cover the same subject.

Mark the decision

Flag URLs that are obsolete, duplicated, thin, orphaned, login-restricted, or generated from near-identical templates. An orphaned page has no internal links pointing to it. A sitemap entry helps a search engine discover a URL; it does not give readers a navigable route to the page.

For every flagged URL, record the evidence and proposed action: improve it, merge it into a stronger page, redirect it to a relevant replacement, exclude it from indexing, or retire it. Check the page’s purpose and owner before removing or redirecting it. A page with little traffic may still support a necessary customer task, such as account recovery or product setup.

Prioritize URLs with customer or business value and a specific observed fault. A product-setup page with no internal links deserves attention before a harmless warning on an unused utility URL. Do not treat every indexed URL, traffic decline, or crawl warning as an implementation defect. Confirm the page’s intended role and the evidence before assigning work.

Name the person responsible for factual accuracy on each page. Name a technical owner when a shared template, routing rule, canonical tag, or access setting requires correction. Clear ownership prevents outdated claims from remaining unresolved.

Check access and indexing

A page may load in a browser yet remain unavailable to a search crawler, excluded from search results, or served with unintended indexing instructions. Check important URLs directly, then compare the response with Google Search Console’s URL Inspection report. A successful browser or curl request confirms only the response received by that request.

Check crawling and indexing separately

A page-level noindex directive asks crawlers that fetch a page not to include it in search results; robots.txt tells matching crawlers which paths they may crawl. If robots.txt blocks a page, crawlers cannot read its noindex directive. Google documents the distinction in its robots.txt guidance and robots meta tag documentation.

For important pages, inspect a representative sample:

HTTP status and redirects
/robots.txt rules that apply
HTML robots meta tags and X-Robots-Tag response headers
Canonical link
Search Console URL Inspection results

A canonical link identifies the preferred URL among duplicate or closely similar pages. It should point to the intended version.

Check both locations that carry indexing instructions. HTML may include <meta name="robots" content="noindex">, while a server, CMS, or CDN may add X-Robots-Tag: noindex to the response header. Headers also apply to non-HTML files, including PDFs. When directives conflict, Google follows the most restrictive applicable instruction. Remove an unintended directive at its source rather than adding another directive to offset it.

Save the response you receive

Save each URL's headers and HTML:

curl -sS -D headers.txt -o page.html https://example.com/important-page/

Check headers.txt for the status line, Location, and X-Robots-Tag headers. Check page.html for the title, robots meta tag, canonical link, and key page text. This command does not follow redirects. If it receives a redirect, inspect the destination URL separately.

If the saved response differs from the intended page, investigate redirects, access rules, CDN behavior, or server output before changing page content. Check server and web application firewall logs for verified crawler requests that receive 403, 429, challenge pages, or server errors. A user-agent string does not prove a request came from Googlebot; use Google’s crawler verification procedure when reviewing logs.

In Search Console, inspect the URL’s reported indexing status. Use a live test when you need Google’s current fetch result. The URL Inspection tool shows Google’s known version of a URL and can test the live URL. Neither result guarantees indexing.

Limit snippets without removing the page

A snippet is the preview text Google may show below a search result. Google’s nosnippet, max-snippet, and data-nosnippet controls limit that preview without asking Google to remove the page from its index.

nosnippet prevents Google from showing a text snippet. max-snippet sets the maximum number of characters Google may use in a snippet. data-nosnippet marks specific HTML content that Google should exclude from snippets. Use an accessible noindex directive when the goal is to remove the page from Google Search, then wait for Google to recrawl it.

Check sitemap mismatches directly

A spot check on November 11, 2026, found that /resources/ and a sitemap-audit page on vincentschmalbach.com returned HTTP 200, used self-referencing canonicals, and had no noindex meta tag. Fifteen selected visible paragraphs longer than 60 characters from the audit page also appeared in the unauthenticated HTML response.

The check also found 827 immediate <url> elements in the urlset served at /sitemap_index.xml, while robots.txt advertised /sitemap.xml and /de/sitemap.xml. These results describe the tested responses at that time. They do not establish that a provider crawler received the same response or indexed either page.

Confirm that every sitemap URL advertised in robots.txt returns valid XML in the intended sitemap format. Compare listed URLs with the site’s intended canonical URLs. Google treats sitemaps as discovery signals, not promises to index every submitted URL, as its sitemap documentation states.

Fix crawler access failures and unintended noindex directives on important pages first. Then correct wrong redirects, canonicals, and sitemap mismatches. Record the observed response, intended behavior, responsible configuration, and later Search Console result.

A successful request from a person or audit script verifies the exact response returned to that request. It remains useful evidence, but it does not establish that Google or another provider crawler fetched, rendered, or indexed the URL.

Separate crawler roles

A site owner can allow a provider to use public pages for search while declining use for model training. Make that decision by crawler role and the provider’s stated purpose. A broad label such as “AI bot” hides differences that affect licensing, traffic, search visibility, and business policy.

Google Search and Google-Extended

For Google Search, including AI Overviews and AI Mode, Google uses Googlebot’s existing crawl and indexing controls. Google’s AI features documentation says these features follow Google Search’s technical requirements. Google-Extended is a separate robots.txt user-agent token.

Google describes Google-Extended as a control over whether site content may help improve Gemini Apps and Vertex AI generative AI models. It does not affect inclusion or ranking in Google Search, including Google’s AI search features.

A publisher that wants Google Search visibility while declining the uses covered by Google-Extended needs separate rules. A Google-Extended directive does not replace Googlebot rules. Check Google’s current documentation before changing policy because product scope and controls change.

OpenAI’s separate controls

OpenAI’s bot documentation assigns distinct purposes to three user agents:

OAI-SearchBot crawls sites for ChatGPT search surfacing.
GPTBot crawls content that may be used to train generative AI foundation models.
ChatGPT-User fetches pages after a user acts in ChatGPT or a Custom GPT.

OpenAI documents OAI-SearchBot and GPTBot as independently configurable. A publisher can allow OAI-SearchBot while disallowing GPTBot, allowing ChatGPT search surfacing while declining the training use OpenAI describes for GPTBot. The appropriate choice depends on the publisher’s content rights, license terms, and commercial interests.

ChatGPT-User requires a separate decision. OpenAI describes it as user-initiated rather than automatic web crawling and says robots.txt rules may not apply to those requests. OpenAI also says the ChatGPT-User user agent does not determine whether content appears in ChatGPT search. A rule for an automated crawler therefore does not control every fetch that follows a user request.

These roles describe OpenAI’s named products and user agents, and establish no rules for other providers.

Broad AI-bot blocks combine different policies

Blocking every crawler identified as AI combines separate uses under one rule. Such a block can remove a site from a search-surfacing path when the publisher’s concern is model training. It also does not resolve whether a provider’s user-triggered retrieval follows robots.txt.

Use each provider’s documented user agent and stated purpose. Do not assume similarly named crawlers share the same role, or that Google’s and OpenAI’s controls apply to another provider.

Use crawler rules for crawler policy

robots.txt communicates instructions to crawlers that choose to follow the protocol. It does not authenticate visitors, authorize access, or protect confidential material. The Robots Exclusion Protocol standard states that robots.txt is not access authorization.

Keep private pages behind authentication and authorization controls. Do not rely on a crawler rule to protect customer records, unpublished documents, paid content, or internal services.

For each provider, decide whether it may crawl for search, model training, or a user-requested fetch. Apply its documented control to the relevant crawler role.

Verify crawler access

An HTTP 200 response proves only what one request received. To investigate a suspected crawler block, check the live robots.txt file. Then match the URL and time of the suspected visit against CDN and hosting logs, as well as firewall and web application firewall (WAF) logs. A WAF screens requests for security threats and can block or challenge traffic before it reaches the origin server.

curl -sS -D - https://www.example.com/robots.txt

Check for rules that apply to the crawler under investigation. Then review the matching request’s source IP address, response status, and recorded security action. A challenge page, rate limit, connection failure, or upstream server error stops access even when robots.txt permits the URL. If a CDN or another intermediary handled the request but does not retain logs, the available records may not show the full response path.

Verify the requester

A requester identifies itself with a user-agent, but anyone can copy a provider’s bot name, so a matching user-agent does not prove the request came from that provider. For example, this command sends a Googlebot user-agent:

curl -sS -D - -o /dev/null \
  -A 'Googlebot' \
  https://www.example.com/page/

The command shows how the server responds to a request claiming to be Googlebot. It does not verify Googlebot’s identity or prove that Google received the same response.

To verify a Googlebot request in your logs, follow Google’s crawler verification procedure: reverse-lookup the logged source IP and check that the returned hostname ends in googlebot.com or google.com. Then forward-lookup that hostname and confirm it returns the same IP address.

host <logged-source-ip>
host <returned-google-hostname>

Use the actual source IP from the logs. Documentation-only addresses, such as 192.0.2.1, cannot verify a real request.

For other providers, follow their current verification instructions. OpenAI’s bot documentation, for example, distinguishes its crawlers and fetchers and links to published IP ranges. Compare the logged source IP with the current range for the specific bot. Do not rely on a copied user-agent, an old IP-range list, or a hard-coded user-agent version.

A simulated request has limited value

A request made with curl remains useful for checking whether a server currently returns a page, redirect, error, or challenge to a particular user-agent. It cannot reproduce the provider’s source IP, request rate, network path, or security reputation. CDN and WAF rules use those details, so a successful simulation does not rule out a block of the real crawler.

Report only what the logs establish

If the available logs do not verify a provider request, describe the result as an unverified access test rather than a confirmed crawl. Missing evidence does not prove that no crawl occurred, especially when CDN, firewall, or origin logs are incomplete or expired. A successful request from a browser or curl does not establish provider access, indexing, citation, or recommendation.

If a verified request received a block or challenge, give the CDN or WAF operator its timestamp, source IP, URL, and response. Include the request or rule identifier if available. Ask which rule handled the request. Change only that rule, then check later logs for a verified request and its response. Broad exceptions based only on a bot name weaken protections without resolving a verified provider failure.

Inspect HTML and rendering

For each important page type, compare the HTML returned by the server with the page after a browser runs JavaScript. Check the main text, page title, canonical link, robots directives, internal links, image URLs, product facts, and structured data. A browser can display information absent from the initial response, and scripts can remove or replace information the server sent.

Compare the response with the rendered page

For each important page type, save an unauthenticated response from a representative URL:

curl -sS https://example.com/important-page/ -o response.html

When you also need the HTTP status and headers, including any X-Robots-Tag directive, save them separately:

curl -sS -D headers.txt -o response.html https://example.com/important-page/

Inspect response.html for the selected text and metadata, then compare them with the rendered DOM in browser developer tools. Check whether readers can see each element; an element can exist in the DOM while remaining hidden.

Record the URL, time, authentication state, and exact content selected for comparison. Keep those conditions consistent across responses. A logged-in comment prompt, location-specific price, or personalized element otherwise looks like a rendering discrepancy when it is simply a different response.

When an essential paragraph, link, or product fact appears only after JavaScript runs, identify the script that adds it and the API or data source that provides it. Put essential text and links in the server response through server-side rendering or static generation where practical. JavaScript can still power interactive features.

Also check whether JavaScript changes or removes the initial title, canonical link, meta robots directive, internal links, product facts, or visible text. Resolve mismatches that change the page’s meaning, indexing instructions, or useful navigation.

Google states that its Search systems process JavaScript and recommends URL Inspection to check Google’s view of a URL. That documentation describes Google’s behavior, not that of every crawler.

A dated spot check

On November 11, 2026, a fresh unauthenticated response for one Article Lab page included all 15 selected visible paragraphs longer than 60 characters outside the comment form. A parser found every selected paragraph in the normalized response HTML without running scripts.

The first comparison included a logged-in comment prompt. It was excluded because an anonymous reader did not receive it. The result describes one page response at that time. It does not establish crawler access, indexing, citation, ranking, or a business outcome.

On the same date, /sitemap_index.xml returned XML with a urlset root and 827 immediate url elements. The XML structure identifies the file as a URL set, regardless of its filename.

A rendering difference is not always a defect

Some differences serve a reader-specific purpose. A page may show an account control for signed-in visitors, a delivery estimate based on location, or an interactive widget after the browser receives visitor data. Treat these differences as expected only when they do not remove essential page information or alter crawler-facing instructions.

Prioritize discrepancies that hide core text or remove internal links. Also prioritize changes to product facts, canonical directives, or robots directives. Record the exact mismatch, then correct the template, script, or API response responsible for it.

Fix URL and canonical errors

Canonical and URL errors send search engines conflicting signals about which pages to show. A canonical link, placed in a rel="canonical" tag, identifies the preferred URL among duplicate or closely similar pages. Google treats the tag as a signal rather than a binding instruction. It also considers redirects, internal links, sitemap entries, and page content when selecting a canonical URL. The canonicalization guidance explains these signals.

Point the canonical link to the intended URL that search engines and visitors can access. Redirects, internal links, and sitemap entries should support that same choice.

Check canonical targets

Inspect canonical links on important page types, especially template-generated URLs and pages with parameters. For example, a product URL containing tracking parameters should generally canonicalize to the clean product URL when both addresses display the same product.

Investigate canonicals that point to the home page, another language version, a redirected URL, a missing URL, a URL blocked from crawling, or a page with materially different content. A translated page should generally canonicalize to its own language version. Use hreflang annotations to identify equivalent pages in other languages, following Google’s guidance on localized versions.

A canonical tag is not a general way to reduce indexed URLs. A filtered category page that shows a substantially different product set may deserve its own URL. Canonicalizing that page to a broader category tells search engines the two pages are duplicates even when they serve different searches.

Correct the template rule or source data that generated the wrong canonical, then test several URLs governed by that rule.

Test redirects and responses

Run this command to see the final status code and destination after curl follows redirects:

curl -sS -L -o /dev/null -w '%{http_code} %{url_effective}\n' https://example.com/old-path/

To inspect every response in the redirect chain and its Location header, run:

curl -sS -L -D - -o /dev/null https://example.com/old-path/

Update internal links to point directly to the final URL, removing unnecessary redirect hops for visitors and crawlers.

A working page should return a successful response and its intended content. A permanently moved page should redirect to the closest relevant replacement. Use a temporary redirect only while the move remains temporary. When no relevant replacement exists, return 404 Not Found or 410 Gone rather than sending the URL to an unrelated page, such as the home page. Google’s documentation covers redirects and HTTP status codes.

The commands show the response received by your request at that time. They do not prove how every search crawler followed, rendered, or interpreted the URL.

Find recurring failures

Use a site crawl to find internal links that lead to missing pages, redirect chains, and server errors. Review web server, CDN, and firewall logs for repeated 4xx and 5xx responses on URLs that should work. Fix internal links to missing URLs, restore pages removed by mistake, and investigate failures at the origin server or an upstream service.

For Google-specific evidence, use Search Console’s URL Inspection tool to check the canonical URL Google selected and its reported indexing status. Other search providers may evaluate canonical and redirect signals differently.

Validate XML sitemaps

An XML sitemap lists URLs a site owner wants search engines to consider indexing. Start with the sitemap URLs advertised in the live robots.txt file. Confirm that each URL returns a sitemap, contains valid XML, and lists the site’s preferred indexable URLs.

Fetch and parse each advertised file

A site may advertise several sitemaps, including separate files for language versions. Fetch each advertised URL to inspect its contents instead of relying on the filename.

curl -sS https://example.com/sitemap.xml -o sitemap.xml

Inspect the response status, final URL, and content type as well:

curl -sS -L -D sitemap-headers.txt \
  -o sitemap.xml \
  -w 'HTTP %{http_code}\nContent-Type %{content_type}\nFinal URL %{url_effective}\n' \
  https://example.com/sitemap.xml

Check sitemap-headers.txt, then parse the saved body with a standard XML parser. A sitemap index has a <sitemapindex> root element and references child sitemap files. A URL sitemap has a <urlset> root element and lists page URLs. The Sitemaps protocol defines both structures.

python3 - <<'PY'
import xml.etree.ElementTree as ET

root = ET.parse("sitemap.xml").getroot()
name = root.tag.rsplit("}", 1)[-1]
child = {"urlset": "url", "sitemapindex": "sitemap"}.get(name)

print("Root:", name)
print("Entries:", sum(
    1 for item in root
    if child and item.tag.rsplit("}", 1)[-1] == child
))
PY

A 200 response and an XML content type do not establish that the file is a usable sitemap. If parsing fails, or the root element is html, inspect the saved body. The server may have returned an HTML error page, a redirect destination, or another unexpected document. If the root differs from the filename, use the XML structure to classify the file.

For a sitemap index, fetch and validate every child sitemap. A redirected sitemap remains valid only if its final URL returns the intended XML document.

List only the preferred indexable URLs

Each sitemap entry should name the canonical URL: the page version the owner wants search engines to treat as the primary result. Remove URLs that redirect, return errors, contain a noindex directive, are blocked from crawling, or duplicate another listed canonical URL.

For an international site, include the intended URL for each language or regional version. Keep sitemap entries aligned with the site’s hreflang annotations, which identify equivalent pages for different audiences. Stale sitemap entries point crawlers to pages that no longer represent the preferred site inventory.

Google recommends listing canonical URLs in sitemaps. Submitting a sitemap helps Google discover URLs, but does not guarantee crawling, indexing, or canonical selection. Submit the file through Google Search Console or the relevant search engine’s webmaster tools, then investigate reported fetch and parsing errors.

Check what the server actually returns

An Article Lab observation dated November 11, 2026, shows why sitemap filenames need checking. On vincentschmalbach.com, /sitemap_index.xml returned a <urlset> with 827 immediate <url> elements, although the path suggests a sitemap index. The site’s robots.txt advertised /sitemap.xml and /de/sitemap.xml.

That observation applies only to the responses recorded on that date. It does not identify the CMS or plugin that produced the structure. Fetch the sitemap URLs advertised in robots.txt, inspect their root elements, and count their direct entries before drawing conclusions from their names.

A valid file still does not ensure indexing

A well-formed sitemap does not override page-level signals. Search engines still assess crawl access, canonical tags, duplicate content, page quality, and indexing directives. Correct a sitemap generator that produces stale entries, but do not remove a valid page solely because one request temporarily failed.

Make pages answer real needs

A product, SaaS, or service page should give readers the facts needed to decide. State what the offer does, who it serves, its public price, requirements, availability, and limits. Link material claims to the product documentation, policy, or terms that support them.

Put decision facts on the page

Name each product's model and variant, then list the specifications, compatibility details, and availability that affect a purchase. Explain warranty, shipping, and return terms, including relevant exceptions. Keep these details consistent with current product feeds and documentation. When availability or terms vary by location or variant, state the condition.

For SaaS, make plan differences easy to compare. State which features each plan includes. Explain its usage limits and supported platforms. Also identify its integrations and prerequisites. Describe data handling accurately and link to current privacy or security documentation. If a feature requires a higher plan or setup work, say so beside the feature.

Service pages need the same precision. Describe the service area, eligibility, scope, process, and exclusions. Tell readers how to confirm current terms. When legal or regulatory rules vary by jurisdiction or change over time, name the jurisdiction and effective date.

Test the page with practical questions: Can someone new to the offer tell which plan fits, what it costs, what they need before buying, and what the offer excludes? If not, improve the explanation or link to the exact supporting information.

Create pages for distinct tasks

Create a new URL when it answers a distinct reader question or supports a different decision. A separate page for a product variant helps when that variant has different specifications, compatibility, availability, or terms. A page that only swaps a place name or keyword and repeats the same advice adds little, so merge near-duplicate pages unless their differences change what readers need to know.

Use headings to group related details. Use lists for requirements or steps, and tables when they make meaningful differences easier to compare. No fixed word count or content block makes a page useful on its own.

Keep visible content, page metadata, and structured data aligned. Google’s AI features guidance emphasizes crawlable pages, internal links, text versions of important content, and consistency between structured data and visible text. Mark up only facts shown on the page and verified by the business. Accurate markup helps search systems interpret the offer; it does not guarantee placement in a search feature or AI-generated response.

Review decision-critical facts when a product, plan, policy, or service term changes. Current, specific information helps readers act and gives search systems a clearer account of the offer.

Keep claims verifiable

For each decision-critical claim, identify the documentation that supports it and the person responsible for updating it. A pricing claim should point to the current pricing page. A compatibility claim should point to the relevant product or support documentation. Record when someone last checked the claim and what triggers another review, such as a plan revision, policy update, or product release.

For product pages, compare visible details with shopping feeds and structured data. Google’s product data specification sets requirements for information submitted to Merchant Center. When price, variant, or availability changes, update the page and feed together. Structured data should match facts readers can see.

For SaaS claims about integrations, supported platforms, or usage limits, link to the exact documentation or policy that supports each statement. Do the same for claims about data handling. Avoid claims such as “works with every platform” when the documentation lists specific versions or requirements. If a claim depends on a location or contract, state that condition and show readers where to confirm current terms.

When another website publishes incorrect information about an offer, record the page, the inaccurate statement, supporting evidence, and any correction request. Use the product owner’s current documentation or policy as evidence. A correction request records the discrepancy; the other site may decline to change its page.

Key takeaway: Connect each important claim to current evidence and a clear owner.

Keep detail usable

Listing every specification on the main page can make a choice harder. Put decision-changing facts where readers will find them, then link to deeper documentation for implementation details, full terms, or uncommon cases. Use a comparison table when readers need to weigh a few meaningful differences. Use prose when conditions need explanation.

Does every customer question need its own page?

No. Add a page when the question represents a distinct task or decision and needs a durable answer. Add a minor clarification to the relevant page when it does not justify a separate destination.

Should a page cite outside sources?

Link to outside documentation when it supports a specific claim, such as a compatibility requirement or legal rule. For claims about a business’s own price, service scope, or product behavior, use current official documentation as the primary evidence. Keep third-party comparisons factual, dated, and tied to the source’s published information.

Connect related pages

Give every important page a useful route from another page on the site. Add a link when the destination helps the reader complete the next part of a task. A product page might link to setup instructions, subscription plans, troubleshooting steps, and the relevant privacy or data policy. A troubleshooting page should link to the documentation or setup step that resolves the problem.

Add return paths when readers need to move between related tasks. Someone comparing plans may need to revisit product features. Someone following a setup guide may need troubleshooting instructions after a failed step. Each important page needs at least one useful internal path so readers and search crawlers can find it.

Use anchor text, the clickable words in a link, to describe the destination or action. Write “compare subscription plans” or “configure single sign-on” instead of repeating “read more” or “click here.” Descriptive anchor text helps readers decide whether to follow a link and helps Google understand the linked page, as explained in Google’s guidance on crawlable links.

Connect product pages, setup instructions, troubleshooting content, comparisons, and relevant policies with a few clear paths. Link because the destination answers the reader’s next question. Avoid large blocks of loosely related links added only to suggest topical connections. They make navigation harder to scan and bury the links that matter.

Check that important links appear in the HTML response and point directly to a working final URL. For a quick check, save the page response and list standard links with href attributes:

curl -sS https://example.com/product/ -o page.html
grep -oE '<a[^>]*href="[^"]+"' page.html

The command misses some valid HTML patterns and links added after JavaScript runs. Use a site crawler for a fuller check. Its report should show the source page, anchor text, destination URL, HTTP status, and redirect target. Update internal links that pass through unnecessary redirects. Repair links that return errors, or decide whether the destination still deserves to exist.

A crawler finds only pages reachable from its starting URLs and permitted by its settings. Compare its results with the XML sitemap, a CMS export, and URLs found in analytics or Google Search Console. A URL that appears in one of those sources but not in the crawl needs investigation. It may be intentionally private, blocked from crawling, available only through on-site search, or missing a useful internal link.

For a page without a clear path, decide what readers need. Add a contextual link when the page serves a distinct task. Merge it into a stronger page when the content substantially overlaps. Remove it when it no longer serves a reader or business purpose. Repeat these checks after site migrations and major navigation changes.

A limit on internal linking

Navigation menus, footers, and related-content modules help readers discover pages at scale. They do not replace contextual links placed where readers need the next step. A setup guide linked only from a large footer remains harder to find and less clearly connected to the product task it supports.

Use structured data carefully

Structured data labels facts on a web page in a machine-readable format. Use it when a Schema.org type accurately describes the page and each marked-up fact is current, verifiable, and available to readers on the page.

Match the type to the page

Choose a Schema.org type that describes the page’s main content. A product detail page may use Product; an editorial story may use Article. The markup should describe the content readers find rather than a search appearance the publisher wants to trigger.

Check every property against the visible page and its supporting business record. Pay particular attention to facts that change over time, including names and prices. Check availability and dates too, along with authorship. Remove values that are outdated, unknown, or unsupported. Google’s structured data policies require markup to represent the page and prohibit misleading, hidden, and irrelevant content.

Google recommends JSON-LD and supports Microdata and RDFa for eligible search features. Where possible, generate the markup from the product, inventory, or publishing fields that also update the visible page. Otherwise, a price change or out-of-stock status can leave incorrect information in the page code.

Validate markup and feature eligibility separately

Use the Schema.org Validator to check whether the markup follows Schema.org syntax and vocabulary rules. If the goal is eligibility for a Google search feature, read that feature’s documentation and test the published or rendered page with Google’s Rich Results Test.

Schema.org validity does not mean Google uses a property for a particular search display. Passing the Rich Results Test means the page is eligible for the tested rich-result type when it meets Google’s requirements; it does not ensure Google will show that result.

Fix errors in the markup and correct the underlying page facts. Treat warnings as prompts to investigate, not instructions to add more fields. Add a suggested property only when it applies to the page and the business can verify it. After publishing, inspect the live URL and relevant Search Console reports. A test against draft code does not confirm what Google received from the live page.

Unsupported markup creates inaccurate claims

Adding every available property creates risk when the page cannot support the claim. A retailer should not mark an estimated delivery date as a confirmed offer, and a publisher should not add an author when no person or organization authored the page. Invented, overstated, or stale markup gives search engines conflicting information and can violate Google’s policies.

Structured data also has a limited role in AI search features. Google states that AI Overviews and AI Mode have no additional technical requirements beyond the requirements for appearing in Google Search. Accurate markup helps search systems interpret supported information in contexts that use it. It does not guarantee rankings, rich results, citations, or inclusion in an AI-generated answer.

Use HTML for its intended purpose

Use headings to organize the document, links to take readers to another resource, lists to group related items, and tables for information arranged in rows and columns. These elements give readers and assistive technologies meaningful structure.

Do not select an HTML tag because someone claims it automatically improves AI visibility. Use the element that represents the content: a heading introduces a section, a table compares tabular facts, and a link leads to a useful destination.

Match the Schema.org type to the page, mark up only facts readers can see and the business can verify, and update the markup when those facts change. Validate Schema.org syntax and Google feature eligibility separately. Structured data supports interpretation; it does not promise a search display or AI citation.

Does every page need structured data? No. Add it when a suitable type accurately describes the page and the publisher can keep the data current. A page without structured data is still eligible to appear in ordinary Google Search results.

Should you add fields because a validator suggests them? Add a field only when it applies, the page supports it, and the underlying fact is current. A validator checks markup structure; Google’s feature documentation sets separate requirements for search displays.

Correct external information

A third-party page with a wrong price or address misleads customers, even if the business’s own website is accurate. Incorrect product details or support instructions do the same. Correct the sources that customers and relevant services actually consult, then record the claim, supporting evidence, and result.

Start with customer-facing sources

Check the external pages that influence a customer’s decision or appear in relevant searches. These include business listings and reseller catalogs, along with partner directories and review platforms. Also check specialist comparisons and community discussions.

Prioritize errors with a direct customer consequence: an incorrect location, an outdated plan limit, a discontinued product shown as available, or unsafe setup instructions. Address low-impact wording differences later when they do not change what a reader understands or decides.

Google explains how owners can find and manage a Google Business Profile through Google Search or Maps while signed in to the associated account. Confirm that you control the right profile before editing it. Other publishers set their own correction procedures, so use the account, editorial, reporting, or support channel the source provides.

Make a specific correction request

Record the exact statement and page URL, along with the date you checked it and the correct information. Include supporting evidence and name the person or organization responsible for follow-up. Useful evidence includes a current product specification, an official policy page, a signed partner agreement, or a dated notice of changed hours.

Ask for a specific factual change and link directly to the evidence. If a software comparison lists an outdated plan limit, identify the incorrect limit and point to the current plan page. Update the same error on your own website first if it appears there.

A publisher has a reason to amend a verifiable factual error. An unfavorable but fair opinion does not establish a factual error and does not justify a removal demand. A reviewer’s statement that a product feels difficult to use expresses an opinion; saying it lacks a documented feature makes a claim that can be checked.

Some platforms display business information that the business cannot edit directly. Report the specific error through the platform’s reporting or support channel rather than sending a vague complaint. Google states that profile edits can take up to three days to appear in Search, so check the public listing after that period to confirm whether the submitted edit is live.

Keep the record honest

Track whether the public page changed and when you verified the change. The record confirms that the source now shows accurate information. It does not prove that search visibility, referral traffic, sales, or customer trust improved.

Do not answer inaccurate external information by inventing reviews, awards, customer accounts, endorsements, citations, or independent comparisons. The Federal Trade Commission’s consumer reviews rule prohibits specified deceptive review and testimonial practices, including fake reviews and testimonials. Label paid or sponsored material clearly. Do not present owned or paid content as independent coverage.

Start with errors that misstate identity, products or services, prices, availability, compatibility, location, safety information, legal terms, or support instructions. Recheck important listings whenever those facts change.

A correction request does not control editorial judgment

A publisher may leave wording unchanged if it accurately states an opinion, summarizes a product differently, or relies on evidence you dispute. Request correction when a claim is factually wrong and provide concise proof. Do not pressure a publisher to remove legitimate criticism or replace independent judgment with preferred wording.

Treat llms.txt as optional

llms.txt is a proposed publishing convention for websites. A site can publish a plain-text, Markdown-style file at /llms.txt to summarize its content and point language-model systems to selected pages. The llms.txt proposal does not define crawler permissions, and no provider is required to request, interpret, or use the file.

Crawler permissions and private-content protection require separate controls. robots.txt gives instructions to crawlers that choose to follow it. Providers publish the user-agent names and controls for their own bots. CDN, hosting, security, and authentication settings determine which requests reach protected resources. Protect private content with authentication and authorization; neither llms.txt nor robots.txt provides that protection.

The case for publishing it

A maintained llms.txt file offers a participating language-model system a short starting point for finding pages the publisher considers useful or authoritative. That benefit depends on the provider choosing to retrieve and follow the file. A file with stale links or obsolete descriptions creates maintenance work without a demonstrated return.

For the Article Lab example, /llms.txt returned a 1,363-byte plain-text response on November 11, 2026. The check confirms only that the URL served a file then. It does not show that an assistant retrieved, interpreted, or used it.

A site can reasonably leave out llms.txt when no one can maintain it or when the site already exposes important content through clear navigation, stable URLs, and accessible pages.

Prioritize by observed problems

Fix confirmed defects on important pages before spending time on optional enhancements. Rank each issue by the importance of the affected URLs, the likely effect on customers or the business, and the evidence that confirms the defect. Adjust that order when a site has a legal deadline, product launch, or other documented priority.

Fix access and indexing faults first

Start with important public pages that return errors, contain an accidental noindex directive, specify the wrong canonical URL, or redirect to an irrelevant or broken destination. Investigate CDN and WAF blocks when logs show that a verified crawler request received a block, challenge, or error.

Inspect each affected URL’s response and headers, including its robots directives and canonical link. Also inspect its redirect destination. Use Search Console to review Google’s reported URL status, and inspect infrastructure logs for access failures.

A robots.txt simulation or a single HTTP 200 response confirms only that test. It does not show that a crawler received the same response, discovered the page, or indexed it.

Correct facts that affect decisions

Next, correct missing or contradictory product and service details, outdated prices or policies, inaccessible key content, and important pages with no useful internal links. Verify each fact against current product records, documentation, or policy. Update every relevant page and feed, then link to important pages from related content that gives readers a reason to visit them.

For an external page, prioritize a false claim that could affect a customer decision, such as an incorrect price, compatibility detail, location, or safety instruction. Record the claim, its URL, evidence supporting the correction, and whether the publisher made the change. A correction request does not guarantee an update or demonstrate an effect on visibility or business results.

Improve content that leaves questions unanswered

After correcting access and factual problems, improve explanations that leave a documented customer question unanswered. Consolidate near-duplicate pages when they serve the same need. Keep separate pages when they support different decisions. Base this work on page content and customer questions rather than a target number of pages.

A dated snapshot of vincentschmalbach.com showed /sitemap_index.xml returning a urlset with 827 immediate URL elements, while robots.txt listed /sitemap.xml and /de/sitemap.xml. A filename does not establish which sitemap the site intends crawlers to use or whether the listed files work. Fetch and inspect each file before deciding what to change.

The same snapshot found 15 selected paragraphs in a fresh, unauthenticated HTML response. That result applies only to those paragraphs and that response. It does not establish that every page exposes its content in the same way or that a crawler could access it.

Leave optional work until later

Refine structured data or publish llms.txt only for a defined purpose, with a named owner responsible for keeping it accurate. Google’s structured data guidance describes supported search features, but a validator checks markup syntax and eligibility signals. It does not show that Google will select the result. Similarly, serving an llms.txt file does not show that an AI provider uses it.

For every change, record the affected URLs and specific defect. Record evidence of the original state, the correction, and a post-change check that confirms the intended implementation. Keep implementation evidence separate from evidence of crawling, indexing, visibility, or business results.

Counterargument: optional work can become urgent

An item stops being optional when a contract, legal requirement, product launch, or documented business dependency requires it. Record that reason explicitly. A deadline alone does not prove that structured data, llms.txt, or another enhancement will improve discovery or revenue.

When GEO needs separate work

Generative engine optimization (GEO) aims to improve visibility in AI-assisted discovery. For most publishers, it does not need separate implementation. Accessible pages and useful, accurate content support both traditional search and AI-assisted discovery. Stable URLs and clear internal links do too. Separate GEO work is justified when a named product documents a distinct control or behavior, or when the publisher needs a different policy for that use.

Key takeaways

Make shared SEO work the default.
Add GEO-specific controls only for a named product, mode, and documented mechanism.
Search eligibility does not guarantee placement or citation in an AI feature.

Google offers a clear example of a separate policy choice. Its Google-Extended documentation describes a crawler control for content used to improve certain Gemini and Vertex AI generative AI products. Google states that Google-Extended does not affect inclusion or ranking in Google Search. A publisher can therefore allow Google Search to crawl and index a page while declining use for model improvement. That choice applies to Google’s stated products and controls; it does not show how another assistant crawls, retrieves, trains on, or selects sources.

Separate decisions also arise when a provider distinguishes automatic crawling from a fetch triggered by a user’s prompt. A live retrieval request can follow different rules from a crawler that discovers pages independently. Treat these uses separately only when the provider documents different controls or behavior; a rule for search crawling does not automatically govern training, live retrieval, or source selection.

Name the product and mode before changing implementation

Before commissioning GEO-specific work, identify the exact product and mode. Check the provider’s documentation for the relevant crawler, access control, or source-selection requirement. Then decide whether the mechanism supports a real content or business policy, and test the change without blocking access the publisher intends to retain.

Google’s statements about Search, AI Overviews, and AI Mode apply to those Google features. They do not establish how every other assistant selects supporting links.

Google’s AI features guidance says a page must be indexed and eligible to appear with a Search snippet before it qualifies as a supporting link in AI Overviews or AI Mode. Google lists no additional technical requirements for that eligibility. The same guidance says AI Overviews appear when Google’s systems determine they add value, and that AI Overviews and AI Mode can show different responses and links. A page must be eligible for indexing to be considered, but eligibility does not guarantee appearance in either feature.

The overlap remains substantial

GEO does not justify a parallel set of generic tactics. Clear explanations, accurate facts, crawlable pages, and sensible site structure remain the work with the broadest value. Create a separate workstream only for verified exceptions: a distinct crawler control, a documented difference between automatic crawling and user-triggered retrieval, or a policy that treats search access and model improvement differently.

Implement a change only when its mechanism is documented or its need is verified. Arbitrary answer-block lengths, unsupported special markup, and claims of guaranteed citations do not provide a reliable implementation plan. Format pages for readers, then add specialized controls when a target product gives a concrete reason.

Keep implementation evidence distinct

Separate implementation, discovery, and outcome evidence when checking search visibility. Implementation evidence shows what the public page or server returned at the time of the check. Examples include a corrected HTTP response header, a canonical link element, or a sitemap entry. A rendered page and firewall rule are also implementation evidence. Discovery evidence shows that a search provider reported a page or that a verified crawler requested it. Search Console indexing status and verified crawler logs fall into this category. Outcome evidence records a defined business result, such as qualified visits, inquiries, sales, or revenue.

Evidence in one category does not prove the next. A self-referencing canonical confirms what the sampled page declared, but does not establish whether Google crawled, indexed, ranked, or cited it. A crawler request shows access, not a sale. A Search Console click shows search traffic, not the cause of an inquiry or the value of a transaction.

The Article Lab snapshot from November 11, 2026 provides implementation evidence only. The tested public pages returned HTTP 200. Sampled pages contained self-referencing canonical link elements, and the checked robots meta tags contained no noindex directive. The selected paragraphs also appeared in the raw HTML returned to an unauthenticated request, without running scripts. These checks describe the pages and responses examined on that date. They do not prove that a search provider accessed or indexed the pages, or that the pages earned citations, rankings, visits, or business results.

Implementation checks still matter because they verify that intended changes reached the public response. Discovery and outcome checks answer different questions and require separate records.

According to Google’s AI features documentation, traffic from its AI features, including AI Overviews and AI Mode, is reported in Search Console’s Performance report under the Web search type. That reporting does not isolate every AI-feature visit, and it does not establish that a technical change caused an inquiry, sale, or other business result. Keep reported visibility, site activity, and business outcomes as separate evidence.

Record the implementation check, discovery signal, and business result separately, including the date, pages or requests tested, and scope of each observation.

SEO and GEO priorities

Start with customer decisions: Use actual questions to choose which pages to improve or create. Record the verified answer, supporting evidence, and useful next step for each.
Audit important URLs: Check status codes, redirects, robots and indexing directives, canonical URLs, sitemap contents, and whether essential text and links appear in the initial HTML. Fix confirmed faults before adding optional features.
Set crawler policies by purpose: Decide separately about search crawlers, training crawlers, and user-triggered fetches. robots.txt gives crawler instructions; authentication protects private content.
Publish accurate facts and useful paths: Keep product, service, and policy details current, then link related pages with descriptive text. Correct significant factual errors on external sites through normal correction channels.
Keep evidence within its limits: A November 11, 2026 site check found 15 selected paragraphs in one page’s response HTML and a urlset with 827 entries at a URL named like a sitemap index. Those observations do not prove crawler access, indexing, citations, rankings, or business impact.

FAQ

Does allowing a crawler in `robots.txt` guarantee it can fetch my pages?

No. A CDN, firewall, web application firewall (WAF), login requirement, or server error can still block or change the response. Check the relevant access logs for the URL and time, then inspect any matching security rule.

How do I check what a page returns?

Run curl -sS -D headers.txt -o page.html https://example.com/important-page/ and inspect the status, response headers, canonical link, robots directives, and important page text. This shows what that request received; it does not prove a search or AI provider fetched the page.

Does a crawler’s user-agent string prove that it visited my site?

No. A requester can copy a provider’s user-agent string, so treat it as a clue rather than verification. For Googlebot, follow Google’s published verification procedure. For other providers, compare current IP information and guidance with server logs.

Should I allow search crawlers but block training crawlers?

That depends on your content and access policy, and on each provider’s controls. As documented by OpenAI, OAI-SearchBot handles ChatGPT search surfacing, GPTBot is associated with possible model training, and ChatGPT-User fetches are triggered by user actions. Check current provider documentation before changing rules.

What if a page is blocked in `robots.txt` but has a `noindex` tag?

A blocked crawler cannot read the page’s noindex tag. If you need a crawler to see that instruction, allow it to fetch the page and return the directive. Use authentication or other access controls for private content.

Does `llms.txt` control crawler access or improve rankings?

No provider documentation establishes that publishing llms.txt controls crawler permissions or improves rankings or citations. Treat it as an optional publishing convention, and use provider-specific controls and site security settings to manage access.

Does valid structured data guarantee a rich result or an AI citation?

No. A validator checks whether the markup is structured correctly; it does not establish eligibility for a particular search feature or whether an AI service selects the page. Keep marked-up facts consistent with what readers can see on the page.

Does qualifying for Google Search guarantee visibility in AI Overviews or AI Mode?

No. Google requires index eligibility and snippet eligibility before a page qualifies as a supporting link, but Google’s systems decide whether an AI Overview adds value and which links appear. Record each exception with the product and mode. Document the behavior, note the policy decision, and name the responsible reviewer.

What should I fix first?

Start with confirmed problems on pages that matter to customers, such as an accidental noindex, a wrong canonical, a broken redirect, a verified access block, or incorrect product or service details. Then address important pages that lack useful internal links or leave real customer questions unanswered.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com