Where `llms.txt` Helps AI Agents, and What It Does Not Prove About Search Visibility

Website owners considering llms.txt for search engine optimization (SEO) need to answer two questions. Can an AI agent, software that fetches and uses web pages to answer requests, use the file to find useful pages? Does publishing it make the site more visible in search results? The file is a public Markdown document that summarizes a site or section and links to selected pages. The proposal gives agents a concise route through a site’s information.

An agent that finds and follows the links can use the file to navigate a large documentation site or find relevant product information. The file alone does not show that a search engine or assistant fetched it, used its links, or will show the site in an answer. Google says it does not use llms.txt for Search, including its generative features, and that the file does not affect Google Search visibility or rankings.

How llms.txt is intended to help an agent

The llms.txt proposal describes a Markdown file that gives an overview of a site and points to relevant material. A documentation site, for example, could use it to direct an agent to its installation instructions, API reference, and troubleshooting pages. The proposal also describes linking to clean Markdown versions of pages where those are available.

The intended path is simple: an agent finds or receives the file, reads its descriptions, chooses a relevant link, and retrieves that page. The file guides the agent to useful material, so it does not have to rely only on menus, page layouts, or its own search through the site.

The file helps only when an agent consults it. The proposal describes a convention, but it does not require commercial agents to look for or follow llms.txt. A clear, up-to-date file can guide compatible tools and human readers, especially on sites with extensive technical documentation. The links need to lead to accurate, usable information.

What it establishes about Google Search

Google’s guidance, checked November 11, 2026, says that Google Search does not use llms.txt or similar special text files for Search, including generative AI features. Google also says that maintaining the file neither helps nor harms Search visibility or rankings. For Google, publishing it is not a requirement for appearing in AI features. See Google’s guidance on generative AI in Search.

Google handles three separate steps: its automated crawlers visit pages, Google adds eligible pages to its index (the collection it can show in results), and Google decides whether to display them. A page must meet Google’s technical requirements to be eligible, but that does not guarantee Google will index or display it. Use Search Console, Google’s tool for site owners, to check important URLs. Review Google’s technical requirements for Search when investigating a visibility problem.

Google’s guidance does not explain how every other assistant or search product handles the file. Providers document controls for their own crawlers, automated programs that visit websites. For example, OpenAI’s crawler documentation distinguishes OAI-SearchBot, which supports ChatGPT Search, from GPTBot, which is associated with potential model training. These controls govern crawler access; they do not show that llms.txt affects search placement or citations. The OpenAI crawler documentation, checked November 11, 2026, also says site owners need to allow relevant traffic in their hosting or content-delivery network settings.

How to check your own site

Request the exact file first, such as https://example.com/llms.txt. Check the response status code and text, and note any redirects. Then open several links in the file. Confirm that they lead to the intended pages, that you can access those pages without logging in, and that the descriptions match the current content.

A successful request from your browser or command line confirms only what happened for that request. A website crawler may receive a different response if a firewall, content-delivery network, rate limit, or bot challenge handles its traffic differently. Review robots.txt and relevant hosting or firewall rules for the crawler you care about. These checks can help you diagnose access, but they do not prove that the crawler visited the file or that a search product used its links.

Check each outcome separately. For Google visibility, inspect important page URLs in Search Console. To assess an assistant’s answers, test questions that matter to your site and record whether it mentions or cites your pages. These results apply to the specific product, prompt, and date you tested. A file request, an indexing status, and an assistant citation each tell you about a different step.

A public web request to vincentschmalbach.com/llms.txt returned HTTP 200, a success status, on November 11, 2026. This confirms that the file responded to that request at that time. It does not show that a provider’s crawler could access the file, that an agent followed its links, or that the site gained search visibility.

Where the file fails to help

A successful response does not guarantee that the file is useful. Its links may lead to outdated pages or broken redirects. They may also point to blocked content or information that conflicts with the pages. A long list without clear organization can make the useful links harder to find. Keep the file focused, describe links accurately, and update it when important pages change.

An agent may ignore the file. If it cannot access the file or use the linked pages, the navigation path does not work. Because the file is public, do not include passwords, private documents, or other information that should not be shared.

Chrome’s Lighthouse documentation, checked November 11, 2026, describes an automated check that tests whether llms.txt can be retrieved. If the file is absent and returns a 404 response, Lighthouse marks the check as not applicable because the file is optional. The check reports whether the file is available. It does not show whether an AI product follows the file or whether search results change. See the Lighthouse llms.txt audit.

Key Takeaways

• Use llms.txt as a navigation aid: Give compatible agents a concise, accurate route to important pages, especially on documentation-heavy sites.
• Separate access from visibility: A successful request proves only that the file responded to that request; it does not prove crawler access, indexing, citations, rankings, or business results.
• Follow Google’s guidance: Google says llms.txt does not affect Google Search visibility or rankings, including for its generative Search features.
• Check each outcome directly: Use Search Console to investigate Google indexing, review crawler controls for the provider you care about, and test assistant answers separately.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com