DevSwitch.net
Sign in
Back to Tools
</>

Sitemap Extractor

Paste a sitemap or site address and extract every listed URL

Web Dev Tools
sitemap_urls.txt

Enter a sitemap URL or a site address that starts with http:// or https://

Works with sitemap index files, nested sitemaps and .xml.gz files. Enter a site address and the sitemaps listed in its robots.txt are used.

Extract URLs from a sitemap or a whole site

This sitemap extractor reads an XML sitemap and gives you every page link in it as one list of URLs. It follows sitemap index files down through every nested sitemap, counts each URL once and sorts the result from A to Z. Copy the list or download it as TXT, CSV or JSON.

You don't need to know where the sitemap is. Enter just the site, such as https://example.com, and the extractor reads the Sitemap: lines in its robots.txt. When robots.txt lists none, it tries /sitemap.xml.

Our server fetches the files, not your browser, so a sitemap on any domain works without cross-origin errors. Only public addresses are fetched: localhost and private network addresses are refused.

Example: a sitemap index with a gzipped sitemap inside

sitemap_index.xml 
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml.gz</loc>
  </sitemap>
</sitemapindex>
sitemap_urls_devswitch.net.csv 
url
"https://example.com/"
"https://example.com/about"
"https://example.com/blog/first-post"
"https://example.com/blog/second-post"
"https://example.com/contact"

The extractor opened both child sitemaps, unpacked the .xml.gz one and returned the page URLs in alphabetical order, under a url header that spreadsheets read as the column name.

What gets extracted and what is skipped

Extracted

  • The <loc> address of every <url> entry
  • Every sitemap listed in a sitemap index, including indexes inside indexes
  • Gzipped sitemaps (.xml.gz) and plain-text sitemaps with one URL per line
  • Relative addresses, completed with the sitemap's own domain

Skipped

  • lastmod, changefreq and priority
  • Image, video and news entries, and hreflang alternates
  • RSS and Atom feeds, which are reported as not a sitemap
  • The pages themselves: nothing is crawled or opened

Limits

  • 25 sitemap files

    One run opens up to 25 files, the index included. For a very large site, extract its child sitemaps one at a time.

  • 50,000 URLs

    The same cap the sitemap protocol sets for a single file. When a run reaches it, the result says so, so you know the list is cut short.

  • 12 seconds per file

    A server that takes longer is listed as failed, and the extractor moves on to the next sitemap.

  • 50 MB per file

    Measured after a .gz file is unpacked, which matches the protocol's own limit.

When one sitemap can't be read, the others still come through. The result counts the failures, such as 3 of 20 sitemaps failed, and lists each file with the reason.

Find the sitemap and extract it

  1. Start with the site address

    Paste https://yoursite.com and the extractor looks in robots.txt for you. That finds the sitemap on most sites, because search engines read the same line.

  2. Or find it yourself

    Open robots.txt and look for a line starting with Sitemap:. Common addresses are /sitemap.xml, /sitemap_index.xml on sites using Yoast SEO and /wp-sitemap.xml on WordPress without an SEO plugin.

  3. Extract and download

    Click Extract URLs, then copy the list or download it as TXT, CSV or JSON. Switching the format rewrites the list straight away, without fetching the sitemap again.

What to do with the URL list

  • Plan redirects for a move

    Save the old site's URLs before a migration and map each one to its new address. The 301 vs 302 guide covers which code each redirect needs.

  • Compare with what Google indexed

    Paste the list and a Search Console export into the diff checker to see pages that are missing from either side.

  • Find broken and moved pages

    The sitemap can still list pages that now redirect or return 404. Run the list through a link checker, such as the one offered under the result, to catch them.

  • Count pages per section

    Sort the CSV in a spreadsheet to see how many URLs each folder holds, and spot test or staging pages that shouldn't be listed at all.

Moving to HTTPS or dropping www?

The .htaccess generator writes the site-wide redirect in one rule. Extract the URL list first to test every page after the switch.

Sitemap extractor questions

A sitemap (<urlset>) lists page URLs, up to 50,000 per file. A sitemap index (<sitemapindex>) lists other sitemaps instead, for sites that need more than one file. Paste either one: the extractor follows an index to every sitemap it names.

Yes. Gzipped files are recognized by their content, whatever the name ends in, and unpacked before reading, up to 50 MB each.

Yes. Enter the site's address or its robots.txt address and every Sitemap: line in the file is read. If there are none, /sitemap.xml is tried.

"Returned a web page" means the address answered with HTML instead of XML, usually a 404 page, a login page or a bot check from a firewall such as Cloudflare. "The server answered 404" means nothing is at that address, so check it in robots.txt. "Not valid XML" points to a broken or cut-off file, "Not a sitemap" to a feed or other XML file, and a size error to a file over 50 MB, which the site should split.

Each URL is counted once, only page URLs are collected (not images), and a run stops at 25 sitemap files or 50,000 URLs. Sitemaps that failed to load are listed under the result with the reason.

No. It only reads the sitemap, so a URL that now redirects or returns 404 still shows up in the list. Run the list through a link checker to test each page.

Not while it is only reachable from your machine or a private network: the fetch runs on our server, which refuses localhost and private IP addresses. Extract it once the site is public, or open the file in your browser and copy the <loc> values from there.