XML Sitemap: What to Include and What to Skip

A search engine can only rank pages it knows about. On a site where every page is well linked, crawlers find most content by following links. On a larger site, a newer site or one that publishes often, some pages can stay undiscovered or be revisited slowly. An XML sitemap is a file that lists the URLs you want search engines to know about, with optional details such as when each page last changed. It does not guarantee anything, but it is one of the simplest technical SEO tools to get right. This guide explains when you need an XML sitemap, what to include and leave out, the limits to respect and how to keep it accurate as the site changes.

What a sitemap does and does not do

Google's documentation explains that a sitemap helps search engines discover URLs on your site, but does not guarantee that every item in it will be crawled and indexed; it adds that in most cases a site will benefit from having one (Google Search Central: Sitemaps overview).

Think of a sitemap as a list of suggestions with supporting detail. Whether a page is indexed still depends on its quality, its links and whether Google considers it the canonical version.

When you need an XML sitemap

The same page says you might not need a sitemap if your site is small, which it describes as about 500 pages or fewer that you want in search results, and comprehensively linked internally. It says you might need one if:

• The site is large, making it harder to ensure every page is linked from at least one other page.
• The site is new and has few external links.
• The site has a lot of rich media content or appears in Google News.

For most business sites with a blog that grows every week, a sitemap is worth having, and most content management systems can generate one.

What to include

Google's guide to building a sitemap recommends including the URLs you want to see in search results, and notes that Google generally shows canonical URLs, which you can influence with sitemaps (Google Search Central: Build and submit a sitemap). In practice, list only pages that:

• Return a 200 status, not redirects or errors.
• Are the canonical version of their content.
• Are not blocked by robots.txt or marked noindex.
• You actually want people to find through search.

The guide also says to use fully qualified, absolute URLs, because Google will attempt to crawl URLs exactly as listed.

What to leave out

Common mistakes come from generating sitemaps automatically without rules:

• Old URLs that now redirect, which send conflicting signals about which page is canonical.
• Duplicate pages, such as several near-identical posts on the same topic.
• Filtered, sorted or tracking-parameter versions of the same page.
• Thin utility pages such as internal search results or login screens.

Cleaning these out makes the sitemap a reliable statement of what matters on your site. Our technical SEO checklist covers how sitemaps fit with redirects, canonicals and internal links.

Format, size limits and sitemap index files

The XML format is defined by the sitemaps protocol, which requires the file to be UTF-8 encoded, begin and end with a urlset tag that specifies the protocol namespace and include a url entry for each page (Sitemaps.org protocol). All data values must be entity-escaped.

Google's guide sets the size limits: a single sitemap may contain up to 50MB uncompressed or 50,000 URLs. Larger sites must split their URLs across several sitemaps and can submit a sitemap index file that points to them.

Use lastmod honestly

Each entry can include a lastmod date. Google's guide says it ignores the priority and changefreq values, but uses lastmod if it is consistently and verifiably accurate. It explains that lastmod should reflect the last significant update to the page, such as a change to the main content, structured data or links, and that a change to the copyright date is not significant.

That means lastmod should come from real content changes, not from the time the sitemap was generated. If every URL shows today's date on every build, the field stops being useful.

Worked example: auditing a blog sitemap

The table below is a hypothetical audit of a company blog's sitemap. It shows the checks worth running and the decisions they lead to.

CheckFindingActionStatus of every listed URLSome URLs redirect to newer postsRemove redirected URLs; list only the targetsDuplicate topicsSeveral posts with near-identical titlesMerge, redirect the extras and list the kept postlastmod valuesAll entries show the build dateUse each post's real last-updated dateAbsolute URLsMix of relative and absolute pathsGenerate absolute URLs on the main domain onlySizeWell under the URL limitNo split needed yet; plan an index file for growthSubmissionNot listed in Search ConsoleSubmit via the Sitemaps report and add a robots.txt line

An audit like this is quick to run with a script that fetches every listed URL and records its status and canonical tag.

Split sitemaps by content type

Even when a site is well under the size limits, separate sitemaps for different content types can make problems easier to spot. A blog sitemap, a services sitemap and a resources sitemap, tied together by a sitemap index, let you see in Search Console which group has errors or pages that are not indexed. If one group shows many excluded URLs, you know where to look first, instead of searching through one long list that mixes everything together.

Tell search engines where it is

Google's guide lists several ways to make a sitemap available: submit it in Search Console's Sitemaps report, which shows when Googlebot accessed it and any processing errors; submit it through the Search Console API; or add a line beginning with Sitemap followed by the full sitemap URL anywhere in your robots.txt file.

Keep it in sync with the site

A sitemap is only useful if it reflects the site as it is now. Generate it automatically from your content system, update it when pages are published, changed, redirected or removed and check it regularly for errors. Fast pages also help crawlers make good use of their visits; our guide to Core Web Vitals covers performance from the user's side.

Frequently Asked Questions

What is an XML sitemap?

It is a file that lists the URLs on your site you want search engines to discover, optionally with the date each page was last significantly updated.

Does a sitemap guarantee indexing?

No. Google's documentation says a sitemap helps discovery but does not guarantee that listed pages will be crawled and indexed.

Should redirected URLs be in the sitemap?

No. List only the final, canonical URLs that return a 200 status. Redirected and duplicate URLs send mixed signals.

Does Google use priority and changefreq?

No. Google says it ignores priority and changefreq, and uses lastmod only when it is consistently and verifiably accurate.

Help search engines find what matters

We audit sitemaps, redirects and canonical tags and set up automatic generation so your sitemap stays accurate. See our technical SEO and content publishing service, or talk to us about your XML sitemap.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.