How Yoast SEO Sitemap Settings Affect Crawling and Indexing

On a WordPress site, Yoast SEO’s sitemap settings determine which XML sitemap files Yoast publishes and which URLs those files list. An XML sitemap is a file that helps search engines discover a site’s URLs. A setting that controls whether content appears in search results can also affect sitemap listings, but sitemap inclusion alone does not make a search engine crawl or index a page.

Your WordPress site serves the sitemap; search engines decide whether to crawl its URLs and add them to search results. After changing a Yoast setting, check both the sitemap output and, when the goal is to keep a page out of search results, the page’s indexing directive.

Sitemap controls and visibility settings do different jobs

Yoast’s XML sitemap feature controls whether Yoast publishes sitemap files. Its sitemap index acts as a directory and links to separate sitemap files for groups such as posts, pages, authors, categories, and tags. Yoast’s current instructions place the feature switch under Yoast SEO → Settings → Site features → Technical SEO; labels and paths can vary by plugin version. See Yoast’s sitemap instructions.

Content visibility settings work at a different level. In the settings for content types, categories, and tags, a maintainer can tell Yoast whether that kind of content should appear in search results. Yoast documents that turning search visibility off also removes the relevant content group from its sitemap. An individual post or page has its own search-visibility control in the editor’s Advanced panel. See Yoast’s settings for customizing the sitemap index.

For example, excluding a content type from search results removes its sitemap group from Yoast’s output. Setting one post to “No” for search visibility excludes that post. These changes differ from switching off Yoast’s sitemap feature, which stops Yoast from providing its sitemap system rather than targeting a particular content group or URL.

Sitemap inclusion does not control crawling or indexing

A sitemap helps search engines discover URLs, especially on large or complex sites. Google treats sitemap submission as a hint: listing a URL does not guarantee that Google will crawl it or include it in search results, as its sitemap overview explains.

A noindex directive tells a search engine not to include a page in its search results. The directive appears in a page’s HTML or in an HTTP response header. Google must be able to crawl the page to read it, so blocking the URL in robots.txt can prevent Google from seeing the directive. Google explains this requirement in its blocking indexing with noindex guidance.

A URL’s absence from a sitemap does not by itself keep search engines from indexing it. When a maintainer turns off search visibility for content in Yoast, Yoast also excludes it from the sitemap. Still, check the page’s noindex directive when the goal is to keep that page out of search results. Likewise, robots.txt controls crawler access; Google warns that it is not a reliable way to keep a URL out of search results, because a blocked URL can still appear without a description. See Google’s robots.txt guidance.

Yoast also says its sitemap files carry an X-Robots-Tag: noindex, follow header. That directive applies to the XML files themselves, so search engines can read the sitemap without indexing the sitemap file. It does not mean that pages listed inside the file receive noindex.

Turning off Yoast may leave another sitemap available

Disabling Yoast’s sitemap feature does not necessarily remove every sitemap from a WordPress site. Yoast says its feature automatically disables WordPress’s built-in sitemap when Yoast’s sitemap is enabled, but turning Yoast’s feature off does not disable the WordPress core sitemap. WordPress documents its own sitemap mechanism separately through the wp_sitemaps_enabled hook.

A sitemap URL that looks familiar does not show which component generated the sitemap. Open Yoast’s View the XML sitemap link and inspect the actual output. If a sitemap still appears after switching Yoast off, check WordPress core for another sitemap. Also check other sitemap plugins. Look for custom code or physical sitemap files.

Verify the output and page directive after a change

Use the following checks to distinguish what your WordPress site serves from what a search engine has processed:

1. Confirm the setting and its scope. Check whether you changed visibility for a whole content type, taxonomy, archive, or an individual URL. Update the post or page after changing its editor-level setting.
2. Inspect the sitemap index and relevant child file. Use Yoast’s view link to open the sitemap. Confirm that the expected sitemap groups and URLs appear and excluded ones are absent.
3. Check the page separately if search exclusion is the goal. Inspect the page’s HTML or its HTTP response for noindex. In Google Search Console, use URL Inspection to see what Google received and review the Page Indexing report for detected noindex directives.
4. Check for conflicting crawl settings and other sitemap providers. Review robots.txt for a crawl block that would hide a noindex directive, and check whether WordPress core or another plugin publishes a sitemap.
5. Rule out stale output and check search-engine processing. If the XML does not reflect the saved setting, check plugin, host, and server caches; Yoast identifies caching as a possible reason for outdated sitemap output in its sitemap troubleshooting guidance. Google Search Console’s Sitemaps report shows whether Google accessed a sitemap and whether it reported processing errors.

Seeing the expected XML in a browser confirms what your site serves; it does not confirm that Google has fetched the file or recrawled an affected page. Google’s sitemap submission guidance describes how to submit and monitor a sitemap, while URL Inspection addresses an individual page’s crawl and index status.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com