How to Separate AI Search Access From Training Permissions

If you run a website, you can let AI search services find its public pages while blocking a provider’s separate training crawler, an automated program that visits and fetches pages. Start with the provider’s rules in robots.txt. The file tells cooperating crawlers which parts of a site they may fetch. It does not secure pages or guarantee that a search service will show them.

You can separate search access from training collection when a provider uses different crawler names or controls for each purpose. Check the rules currently served by each host and use the provider’s current bot names. Also account for other ways the provider can retrieve pages.

How robots.txt separates crawler purposes

A robots.txt file sits at the root of a host, such as https://example.com/robots.txt. A host is a domain or subdomain. The file’s User-agent line names a crawler. Allow and Disallow rules tell that crawler which paths it may fetch. The Robots Exclusion Protocol explains how cooperating crawlers read these instructions. The protocol does not control who can access a page.

For example, a site that wants to permit OpenAI’s search crawler and Anthropic’s search crawler while blocking their named training crawlers could use rules like these:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

These rules are a starting point, not a complete policy. OpenAI identifies OAI-SearchBot as the crawler for ChatGPT search and GPTBot as a crawler that collects content that may be used to train its models. Anthropic identifies Claude-SearchBot as its search crawler and ClaudeBot as a crawler that may collect training data. Both providers describe these roles in their crawler documentation and site-owner guidance.

A provider may also fetch a page in response to a user’s request. OpenAI’s ChatGPT-User and Anthropic’s Claude-User handle some of these requests. Decide whether to allow these fetches as well. Blocking a training crawler does not control every way the provider can retrieve a page. Check the provider’s current rules for each bot before changing your file.

Google uses a different arrangement. Google-Extended is a control token in robots.txt, not the name of a separate crawler sent in HTTP requests. Google says the token affects certain training and grounding uses. It leaves inclusion in Google Search unchanged. To allow Google Search crawling while restricting the uses covered by Google-Extended, add:

User-agent: Google-Extended
Disallow: /

Check Google’s current description of Google-Extended and its other crawlers before treating this as equivalent to another provider’s training-bot rule. Its scope includes specified grounding uses as well as model training.

Check the rules your site actually serves

Open https://your-host.example/robots.txt in a browser, or fetch it with a command such as curl -i https://your-host.example/robots.txt. Confirm that the server returns the intended plain-text file. Read the whole file instead of relying on a content-management system setting. A plugin or hosting provider may control the version visitors and crawlers receive; a content delivery network may do so too.

Check each relevant host separately. For example, example.com and shop.example.com each have their own root-level robots.txt file. For each one, find the bot names documented by the provider. Check their rules along with wildcard rules and repeated groups. Under the RFC, matching groups can combine, so another group’s rules can change how a rule works.

Check the specific path you care about. A rule can allow one part of the site and block another. Page-level noindex instructions and HTTP headers control search indexing; they do not set a crawler’s training permission. Google must be able to access a page to read its noindex instruction, and does not support noindex rules in robots.txt.

On November 11, 2026, a public request to www.vincentschmalbach.com/robots.txt returned a file with a successful HTTP status. A local robots parser treated /resources/ as allowed for OAI-SearchBot and GPTBot under the file’s general rules. That observation describes the file and the parser’s interpretation at that time. It does not show that either crawler fetched the page.

After changing the file, check your server and CDN logs for requests and blocked requests. A user-agent string in a log can help identify a crawler, but it does not prove the crawler’s identity because a request can imitate the string. A successful request from your browser or command line also does not prove that the provider’s crawler can reach the page.

Account for provider differences and developing controls

Providers use different bot names and assign them different purposes, so the same rule does not make the same choice for every provider. Anthropic’s Claude-User, for example, is distinct from its search and training crawlers. Google-Extended also covers uses beyond training. Decide which uses to permit, then match those choices to each provider’s current documentation.

A shared signal for expressing AI-training preferences is also emerging. As of November 11, 2026, Anthropic says it has begun recognizing a signal based on developing IETF work, though the rollout is still in progress. Google has announced plans to support a shared control in 2027. These statements do not show that all providers support a shared control or that its rules are settled. For a current decision, use each provider’s documented crawler controls and check its latest instructions about any shared signal.

Know what the controls cannot do

When your rules allow a search crawler to fetch certain paths, that permission does not show that the crawler visited them, that the service indexed them, or that an AI answer will cite them. Google says it does not guarantee that it will crawl, index, or show a page in Search. OpenAI also says a page can still appear as a navigational link in ChatGPT search when OAI-SearchBot is blocked.

Blocking a crawler in robots.txt does not hide a page from people or stop crawlers that ignore the protocol. If a page must remain private, protect it with authentication or remove it from public access. Google’s robots.txt guidance likewise warns against using the file to protect confidential information.

A new rule does not tell you what a provider does with material it collected before the change. Crawler instructions set access preferences; they do not promise to delete material from existing datasets, model weights, or copies held elsewhere. Treat a change to robots.txt as an instruction for future crawler behavior, not a request to remove material already collected.

Key Takeaways

• Match rules to crawler purpose: Allow the provider’s documented search crawler and block its training crawler when those are separate controls.
• Check every host and matching rule: Inspect the live robots.txt file, including wildcard and repeated groups, for each domain or subdomain.
• Treat Google-Extended separately: Google says it affects training and specified grounding uses without controlling Google Search inclusion.
• Use access controls for private content: robots.txt is a public instruction for cooperating crawlers, not a security barrier.
• Separate permission from visibility: An allowed crawler does not prove a visit, indexing, citation, or appearance in an answer.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com