How to Handle Language Selection Without Forcing Search Crawlers Elsewhere

When a website serves readers in several languages or countries, automatic routing can send a visitor to a different page based on their location, browser language, or a saved preference. That behavior is an international redirect. If it also redirects a crawler away from the URL it requested, the crawler may not see that URL’s intended language version.

Give each language or regional version its own stable URL, and let readers choose another version with a visible link or suggestion. A crawler should be able to request each URL directly, without needing a particular cookie, browser setting, or location. Google recommends separate URLs for locale versions and cautions that automatic redirects can prevent people and search engines from viewing every version.

Let readers choose without rerouting every request

A language selector should link to another version, rather than making that version available only to visitors with a particular IP address or browser setting. For example, a selector on a French page should link to the corresponding English page when one exists. If there is no equivalent page, link to a relevant language landing page.

Keep the selector as an ordinary link that works without JavaScript. After a visitor chooses a language, you can save that preference for later visits, as long as it does not block direct access to other language URLs.

This matters because Google says its default crawler requests do not include an Accept-Language header, which websites commonly use to infer a reader’s preferred language. Google also describes crawling from outside the United States, but warns that it may not crawl, index, or rank every version of a page that changes according to the request. Its guidance for multilingual and multi-regional sites therefore favors separate URLs over changing one URL’s content based only on browser settings or cookies.

Use hreflang to connect equivalent pages

hreflang is an annotation that tells Google which language or regional pages are alternatives to one another. It describes relationships between pages; it does not redirect visitors or make an otherwise inaccessible URL available.

For each set of equivalent pages, include a reference to each page, including its alternatives. Each page should also link back to the other versions. Google supports putting these annotations in page HTML, HTTP headers, or a sitemap. It says using multiple methods provides no Search benefit. An x-default entry can identify a fallback, such as a language-selection page, when no version matches a visitor’s preference. See Google’s instructions for localized page versions.

The page itself still needs substantive content in the intended language. Google says it determines a page’s language from visible content, rather than from hreflang or the HTML lang attribute. The lang attribute remains useful for browsers and accessibility tools, but it does not replace localized content or direct access to the page.

Check what each URL actually returns

Test important language URLs directly, including pages served through a content delivery network (CDN) or another edge service. Those services can redirect a request before the page’s own language selector loads. For each URL, compare a request without cookies against requests with different language headers, preference cookies, or test locations.

Google’s URL debugging tools can show how Google reports a URL in Search and allow a live test. Logs provide another view of requests, but the user-agent string, which identifies the requesting software, alone does not prove that a request came from Google. Neither a live test nor a log entry proves that Google will index or show every language version.

Understand what a redirect and a successful test mean

A 200 response with the expected language confirms that the URL served that content for the request you tested. It does not confirm that Google indexed the page or will show it for searches in every country.

Google follows both permanent and temporary redirects. A permanent redirect, such as 301 or 308, signals that the destination is the preferred URL. A temporary redirect, such as 302 or 307, does not make the original language page visible at the original URL while the redirect remains in place. Changing the redirect type therefore does not solve a routing rule that sends every request to another language. Google explains these distinctions in its guidance on redirects and Search.

Google’s documentation supports using separate URLs because locale-dependent delivery can limit discovery. It does not promise a ranking improvement or quantify how much traffic a particular redirect causes a site to lose. Test results apply only to the URLs and request conditions you tested. A crawler reaching one language page does not show that it can reach every other version.

Keep the same access limits in mind for AI search

Google says pages must be indexed and eligible to appear with a snippet in Search to be eligible as supporting links in AI Overviews and AI Mode. That makes direct access to each intended language URL relevant to those features too. Google specifies no additional technical requirement for these AI features, and URL accessibility does not guarantee that a page will be selected or cited. See its current AI features guidance.

Key Takeaways

• Give every language version a stable URL that visitors and crawlers can request directly.
• Offer language changes as links or suggestions instead of automatically redirecting every request based on IP, browser language, or a saved preference.
• Use reciprocal hreflang annotations to identify equivalent language or regional pages, and make sure each page contains substantive content in its intended language.
• Test the response and redirect chain across relevant URLs, cookies, language headers, and locations; one successful test covers only the conditions tested.
• Treat crawl access as a prerequisite, not a visibility guarantee. A reachable page can still remain unindexed or absent from Search and AI features.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com