Sitemap in robots.txt: a third of Polish files omit it

A third of Polish robots.txt files name no sitemap. We read the robots.txt files of the 6,834 .pl sites in the Tranco list of popular sites (opens in a new tab) that serve one, and 35% have no Sitemap: line. Of those that do, 6% point to an address the site doesn't serve as written: http on an https site, the www twin of the real host, a relative path, a test server.

The line is the one place where every crawler, not just Google, learns where a site's list of pages is. When it's missing, crawlers are left to guess, and the guess often fails: on the largest Polish sites with no Sitemap: line, /sitemap.xml held a sitemap for only 34% of them. Whether the line is there goes with the platform behind the site, and wrong pointers are more common on sites where we recognised no platform than on those where we did. In our study of AI crawlers on the Polish web a sitemap listed where crawlers look for it was one of the basics Polish sites lagged on; this is the full census.

robots.txt files read

6,834

name no sitemap

35%

of those naming one point wrongly

6%

of large sites without the line have /sitemap.xml

34%

What the Sitemap line in robots.txt does

A sitemap is a file, usually XML, listing the pages a site wants crawled, defined by the sitemaps.org protocol (opens in a new tab). A crawler can learn where it is in two ways: the site owner submits it in a search engine's own console (Google Search Console, Bing Webmaster Tools), which reaches that one engine, or the site names it in robots.txt, which every crawler reads first.

RFC 9309 (opens in a new tab), the robots.txt standard, doesn't define the line; it lists Sitemaps as an example of "other records" crawlers may read, and says such a record must not end a User-agent group. The rules come from elsewhere:

• sitemaps.org: the line holds "the full URL to the sitemap", and a file may name more than one. A sitemap's URLs must use its own protocol and host, unless the other host's robots.txt points to it (a "cross submit").
• Google's robots.txt documentation (opens in a new tab) (last updated 31 August 2026): the URL "must be a fully qualified URL, including the protocol and host", it "doesn't have to be on the same host as the robots.txt file", and any number of lines is fine. The line "isn't tied to any specific user agent".

So a correct line looks like this, anywhere in the file:

User-agent: *
Disallow: /cart/
 
Sitemap: https://www.example.pl/sitemap.xml

A third of Polish sites name no sitemap

Of the 7,785 sites whose homepage answered in our census, 951 served no robots.txt as a text file (most answered with a 4xx error, some sent an HTML page instead); they are left out. Among the rest, most sites name a sitemap and point to their own address. The second-largest group names none at all, and the groups that point wrongly are small but not rare.

No Sitemap line35%

Points to its own address59%

Points only to other hosts1.7%

Another protocol or www twin3.1%

Unusable as written0.8%

Source: Outofplace census, 24 Sept 2026; Tranco list Y8YYGDownload CSVDownload PNG

Show the data

Item	Value	95% interval	n
No Sitemap line	35%	34%–37%	6,834
Points to its own address	59%	58%–60%	6,834
Points only to other hosts	1.7%	1.4%–2%	6,834
Another protocol or www twin	3.1%	2.7%–3.5%	6,834
Unusable as written	0.8%	0.6%–1.1%	6,834

The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.

Size doesn't help. Among the sites in the sample's top 1,000, 59% name a sitemap, fewer than the 65% of the sites ranked below them. The gap goes with the mix of software. 74% of the top sites run nothing we recognise as a shop platform, an SEO plugin or WordPress, against 45% of the sites below them. Compared kind with kind, top sites and the rest name a sitemap about equally often: 52% against 55% where we recognised nothing, 76% against 74% where we did.

Should the sitemap be in robots.txt?

It isn't required: a search engine can find pages through links, and Google and Bing accept a sitemap submitted in their consoles. But a console reaches one engine, and every other crawler (Bing for a site only in Google's console, the AI crawlers, SEO tools) has the robots.txt line or a guess.

The AI crawler study fetched one sitemap per site for the largest 1,000 .pl domains on 23 September: the first Sitemap: URL when robots.txt names one, else /sitemap.xml, the conventional default. The two show what a crawler finds with the line and without it.

Named in robots.txt: the first Sitemap URL96%

Not named: a guess at /sitemap.xml34%

Source: Outofplace AI crawler study crawl, 23 Sept 2026; Tranco list Y8YYGDownload CSVDownload PNG

Show the data

Item	Value	95% interval	n
Named in robots.txt: the first Sitemap URL	96%	95%–98%	511
Not named: a guess at /sitemap.xml	34%	29%–39%	362

The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.

96% of the named URLs answered with a sitemap. Without the line, /sitemap.xml did for 34% of the sites; the rest mostly answered with an error, some with an HTML page. Those sites may well have a sitemap elsewhere, submitted to Google, but a crawler that only reads robots.txt can't know.

Who writes the Sitemap line: the platform or nobody

A CMS or shop platform can serve robots.txt itself, and then whether the file names a sitemap is up to the platform's settings. WordPress 5.5 (opens in a new tab) (2020) made its own robots.txt "reference the sitemap index" at /wp-sitemap.xml; SEO plugins replace it with their own.

IdoSell100%

Shopify98%

Rank Math (WordPress)90%

WooCommerce85%

Headless shop84%

All in One SEO (WordPress)83%

Yoast SEO (WordPress)78%

Custom shop74%

Magento72%

WordPress, no SEO plugin found72%

PrestaShop62%

Other platforms62%

No platform recognised54%

Shoper47%

AtomStore39%

Source: Outofplace census, 24 Sept 2026Download CSVDownload PNG

Show the data

Item	Value	95% interval	n
IdoSell	100%	97%–100%	217
Shopify	98%	94%–100%	127
Rank Math (WordPress)	90%	84%–94%	135
WooCommerce	85%	81%–88%	381
Headless shop	84%	71%–92%	44
All in One SEO (WordPress)	83%	73%–90%	76
Yoast SEO (WordPress)	78%	74%–81%	662
Custom shop	74%	68%–79%	214
Magento	72%	66%–78%	201
WordPress, no SEO plugin found	72%	68%–76%	437
PrestaShop	62%	58%–66%	553
Other platforms	62%	54%–69%	173
No platform recognised	54%	53%–56%	3,351
Shoper	47%	40%–53%	227
AtomStore	39%	25%–55%	36

The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.

Hosted shop platforms sit at both ends. On IdoSell and Shopify nearly every shop names its sitemap. On Shoper and AtomStore about half of the shops in our sample or fewer name one, and the two can't be told apart (AtomStore is a small group of 36 shops). From robots.txt alone we can't tell whether that comes from a platform setting, a per-shop option or files the shops edit themselves. WordPress with an SEO plugin sits high, and the largest group, the sites where we recognised no platform at all, is close to the bottom with 54%.

File names point to platforms too. sitemap.xml is the most common, on 45% of the sites with a pointer to their own host, and some of the next names belong almost entirely to one kind of software, such as 1_index_sitemap.xml to PrestaShop and googlesitemap to Shoper:

How often is the sitemap pointer wrong?

We sorted every Sitemap: value by what it points at, against the address the homepage actually landed on. Pointers to the site's own address are the norm. Pointers to another host are allowed by both Google and sitemaps.org and say nothing wrong on their own: a sitemap on a CDN, a shop's subdomain, a sister site. The rest fall into two groups.

• Mismatched: the site's own host with http instead of https (or another port), or the www twin of the host the homepage lands on. These work only if the server redirects, and by the sitemaps.org rule a sitemap fetched from http:// speaks for http:// URLs.
• Broken: no usable address as written. A relative path such as /sitemap.xml, which Google's rule for a fully qualified URL excludes; a host with no https://; localhost, a .test domain or a dev. host left over from staging; or a template whose placeholder was never filled in ({page_domain}/…, undefined/…).

Another site3%

www instead of the bare domain, or back2.4%

http instead of https (or another port)2.4%

A relative path (/sitemap.xml)1%

Another host of the same site0.7%

localhost, .test or a staging host0.1%

An unfilled template ({page_domain})0.1%

No http(s):// before the host0.1%

Source: Outofplace census, 24 Sept 2026Download CSVDownload PNG

Show the data

Item	Value	95% interval	n
Another site	3%	2.5%–3.5%	4,411
www instead of the bare domain, or back	2.4%	2%–2.9%	4,411
http instead of https (or another port)	2.4%	2%–2.9%	4,411
A relative path (/sitemap.xml)	1%	0.7%–1.3%	4,411
Another host of the same site	0.7%	0.5%–1%	4,411
localhost, .test or a staging host	0.1%	0.1%–0.3%	4,411
An unfilled template ({page_domain})	0.1%	0%–0.2%	4,411
No http(s):// before the host	0.1%	0%–0.2%	4,411

The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.

Together, 266 sites name at least one wrong address: 211 mismatched and 55 broken. That would matter less if the wrong line sat next to a right one, but it rarely does: 247 of them name no sitemap on their own address at all, and for 53 every pointer is broken.

Wrong pointers are more common where we recognised no software. Where a platform, plugin or WordPress was recognised, 4.1% of the sites naming a sitemap point wrongly; where none was, 8.8% do.

Multiple sitemaps in robots.txt

Most files name one sitemap: 85% of the sites that name any. The longest file we found lists 492. Google sets no limit, so a long list isn't an error, but it is a sign the site lists every sitemap where one sitemap index would do.

1 line85%

2 lines9.1%

3–5 lines3.3%

6–20 lines2%

21 or more0.5%

Source: Outofplace census, 24 Sept 2026Download CSVDownload PNG

Show the data

Item	Value	95% interval	n
1 line	85%	84%–86%	4,411
2 lines	9.1%	8.2%–9.9%	4,411
3–5 lines	3.3%	2.8%–3.9%	4,411
6–20 lines	2%	1.6%–2.5%	4,411
21 or more	0.5%	0.4%–0.8%	4,411

The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.

Repeats are rarer: 45 sites list the same URL more than once, 115 extra lines in all. They do no harm, and they suggest a file that grew by copy and paste.

How to add a sitemap to robots.txt

1. Check what your robots.txt says now

   Open https://your-domain/robots.txt in a browser, or from a terminal:

   curl -s https://www.example.pl/robots.txt | grep -i '^sitemap'

   Finding no line at all is common. If your platform writes the file, look for the setting there before editing anything by hand.

2. Find the sitemap's real address

   Open the address your homepage lands on and note its protocol and host: https://example.pl and https://www.example.pl are different hosts to a crawler. The sitemap is usually at /sitemap.xml, /sitemap_index.xml (SEO plugins such as Yoast SEO) or /wp-sitemap.xml (WordPress without a plugin); open it and check that it is XML, not your 404 page.

3. Write one full URL per line

   Add the line anywhere in the file, outside or inside a group: it applies to every crawler either way.

   Sitemap: https://www.example.pl/sitemap_index.xml

   Not /sitemap.xml, not www.example.pl/sitemap.xml, not http:// on an https site.

4. Prefer one sitemap index to many lines

   Under the sitemaps.org protocol (opens in a new tab), a sitemap holds up to 50,000 URLs and 50 MB uncompressed; past that, split it and list the parts in a sitemap index, which may list 50,000 sitemaps. Name the index in robots.txt. A sitemap on another host (a CDN, a subdomain) is fine when your robots.txt points to it.

5. Check it after every migration

   Moving to https, adding or dropping www, or copying a staging site to production is when pointers go stale. Fetch robots.txt again after the move and open each Sitemap: URL.

How we measured

Further reading