An XML sitemap is a file that lists the URLs on your site and tells search engines which pages you want crawled and indexed. Robots.txt is a separate file that tells crawlers which parts of your site to stay out of. Together they control a significant part of how Google navigates your site, and on most WordPress installations I audit, both have configuration issues that are quietly causing problems.
Neither file is complicated to set up correctly. The issues I find are almost always the result of plugin default settings that were never reviewed, or manual entries added without understanding their effect. Getting both right is one of the highest-leverage technical fixes on a site that is underperforming for its content quality.
XML sitemaps: what they do and what they should contain
Your sitemap is a signal to Google about what exists on your site and what you consider worth indexing, not a guarantee that those pages will be indexed, and not a ranking signal in itself. Think of it as a map you hand to a visitor rather than a set of instructions they must follow. Google may crawl pages not in your sitemap, and may not index pages that are in it. Its primary value is reducing the chance that important pages are missed.
Like this stuff? Get it weekly.
The Wednesday Roundup. SEO and AI search news, tactics and what is changing this week. No fluff.
Subscribe →The most common sitemap errors I see on WordPress sites: including pages with noindex tags (contradictory, if you do not want a page indexed, do not include it in the sitemap), including redirect URLs (the sitemap should contain canonical destination URLs only), including 404 pages (Screaming Frog frequently flags these on sites that have deleted content without updating the sitemap), and omitting important pages because they were set to noindex by mistake.
WORK WITH JAMES
Get a straight answer on your SEO.
No account managers. No generic reports. Direct access to someone who has been doing this for 25 years.
On Rank Math, the sitemap is at /sitemap_index.xml by default. It generates separate sitemaps for posts, pages, categories, and custom post types, which are linked from the index. Check each child sitemap for the errors above, the issues are usually in the pages or categories sitemaps rather than the posts sitemap.
What to exclude from your sitemap
Exclude any page you would also noindex. This includes admin pages (if somehow accessible), thank you pages and order confirmation pages, login and checkout pages, author archive pages (on single-author blogs these are near-duplicate content), tag archive pages with very few posts, and any page that is a thin variant of another page rather than genuinely distinct content.
The logic: your sitemap is a recommendation list. Including low-quality or thin pages in it tells Google you consider these worth indexing, which dilutes the signal quality of the sitemap overall. A sitemap with 50 strong pages is more useful than one with 300 pages of varying quality.
Submitting your sitemap to Google Search Console
Submit your sitemap URL in Google Search Console under Indexing > Sitemaps. Enter the full URL including the protocol: https://yourdomain.com/sitemap_index.xml. GSC will show when it was last read and whether it processed the URLs without errors.
After significant site changes, migrations, URL restructures, large content additions, request a re-read by removing the sitemap from GSC and resubmitting it. Rank Math also pings Google via IndexNow when individual pages are updated, which is faster than waiting for Google to recrawl your sitemap on its own schedule.
Robots.txt: what it does and the mistakes that cause real damage
Robots.txt is a text file at the root of your domain (/robots.txt) that tells crawlers which areas of your site to skip. Unlike noindex tags, it prevents crawling rather than indexing, a page blocked by robots.txt can still be indexed if other sites link to it, but Google cannot see its content to understand what it is about.
The most damaging robots.txt error is blocking the wrong directories. I have seen production sites where someone added Disallow: / to robots.txt during development and never removed it, effectively asking Google to ignore the entire site. This is hard to diagnose because the site looks normal to users but is invisible to crawlers. If your organic traffic suddenly dropped to near-zero after a site change, checking robots.txt should be your first step.
Other common issues: blocking /wp-content/uploads/ which prevents Google from crawling images (relevant if you want image search traffic), blocking CSS and JavaScript files which prevents Google from rendering your pages correctly, and adding Disallow rules for URLs using old slugs that no longer exist (harmless but messy).
What a clean robots.txt looks like on WordPress
For most WordPress sites, a minimal robots.txt is correct. The standard Rank Math default blocks /wp-admin/ (allowing admin-ajax.php for functionality) and a handful of other admin paths. It includes a Sitemap: directive pointing to your sitemap_index.xml. Beyond that, additions should be deliberate and specific, not copied from a template without understanding what each line does.
Test your robots.txt in Google Search Console under Settings > robots.txt tester. Check any URL you are concerned about to confirm whether it is being blocked. After any changes, monitor GSC’s coverage report over the following weeks to confirm the expected pages are being crawled.
Frequently asked questions
Not directly. A sitemap does not boost rankings. What it does is help Google discover and prioritise pages it might otherwise miss or take longer to find, particularly on large sites or sites with thin internal linking. For most small business sites with under 100 pages, the sitemap is a hygiene factor rather than a ranking lever.
Go to Google Search Console, select your property, navigate to Sitemaps in the left menu, and enter your sitemap URL. For most WordPress sites this is your domain followed by /sitemap_index.xml. You only need to submit it once, Google will re-fetch it automatically. You can also add your sitemap URL to your robots.txt file.
Exclude pages that should not be indexed: admin pages, thank you pages, login pages, duplicate content pages, and any page with a noindex directive. Including noindex pages in your sitemap sends a mixed signal to Google. Rank Math and Yoast both handle this automatically if your noindex settings are configured correctly.
Google can process sitemaps with up to 50,000 URLs. If you have more pages than that, split into multiple sitemap files and reference them from a sitemap index file. For most small business sites the question is less about the limit and more about being selective, only include pages you actively want Google to crawl and index.
Google fetches your sitemap periodically, typically within a few days of submission and then on an irregular schedule thereafter. If you make significant changes to your site structure or publish new content regularly, you can request a re-fetch via Google Search Console by deleting and resubmitting the sitemap URL. Plugins like Rank Math also ping Google via IndexNow when individual pages are updated.
TESTIMONIALS
