---
title: "Sitemap in robots.txt: a third of Polish files omit it"
description: "35% of 6,834 .pl sites with a robots.txt name no sitemap in it, and 6% of those that do point to a wrong address: http, the www twin or a relative path."
canonical_url: "https://www.outofplace.space/blog/sitemaps-robots-txt-polish-web"
language: "en"
last_updated: "2026-10-12"
alternates:
  pl: "https://www.outofplace.space/pl/blog/sitemap-w-robots-txt.md"
---

# Sitemap in robots.txt: a third of Polish files omit it

> 35% of 6,834 .pl sites with a robots.txt name no sitemap in it, and 6% of those that do point to a wrong address: http, the www twin or a relative path.

Published 2026-10-12 · Category: Search & AI · Tags: robots.txt, XML sitemaps, Poland

A third of Polish robots.txt files name no sitemap. We read the robots.txt files of the 6,834 .pl sites in [the Tranco list of popular sites](https://tranco-list.eu/list/Y8YYG/full) that serve one, and 35% have no `Sitemap:` line. Of those that do, 6% point to an address the site doesn't serve as written: `http` on an `https` site, the `www` twin of the real host, a relative path, a test server.

The line is the one place where every crawler, not just Google, learns where a site's list of pages is. When it's missing, crawlers are left to guess, and the guess often fails: on the largest Polish sites with no `Sitemap:` line, `/sitemap.xml` held a sitemap for only 34% of them. Whether the line is there goes with the platform behind the site, and wrong pointers are more common on sites where we recognised no platform than on those where we did. In [our study of AI crawlers on the Polish web](https://www.outofplace.space/blog/ai-crawlers-polish-web) a sitemap listed where crawlers look for it was one of the basics Polish sites lagged on; this is the full census.

- 35% of 6,834 Polish sites with a robots.txt name no sitemap in it.
- Whether a site names one goes with its platform: 99.5% of IdoSell shops do, against 46.7% of Shoper shops.
- 266 sites name a wrong address, and 93% of them give no correct one besides.
- Where no platform or plugin was recognised, 8.8% of the sites naming a sitemap point wrongly, twice the rate elsewhere.

## What the Sitemap line in robots.txt does

A sitemap is a file, usually XML, listing the pages a site wants crawled, defined by the [sitemaps.org protocol](https://www.sitemaps.org/protocol.html). A crawler can learn where it is in two ways: the site owner submits it in a search engine's own console (Google Search Console, Bing Webmaster Tools), which reaches that one engine, or the site names it in robots.txt, which every crawler reads first.

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.4), the robots.txt standard, doesn't define the line; it lists `Sitemaps` as an example of "other records" crawlers may read, and says such a record must not end a `User-agent` group. The rules come from elsewhere:

- sitemaps.org: the line holds "the full URL to the sitemap", and a file may name more than one. A sitemap's URLs must use its own protocol and host, unless the other host's robots.txt points to it (a "cross submit").
- [Google's robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) (last updated 31 August 2026): the URL "must be a fully qualified URL, including the protocol and host", it "doesn't have to be on the same host as the robots.txt file", and any number of lines is fine. The line "isn't tied to any specific user agent".

So a correct line looks like this, anywhere in the file:

```txt title="robots.txt"
User-agent: *
Disallow: /cart/

Sitemap: https://www.example.pl/sitemap.xml
```

## A third of Polish sites name no sitemap

Of the 7,785 sites whose homepage answered in our census, 951 served no robots.txt as a text file (most answered with a 4xx error, some sent an HTML page instead); they are left out. Among the rest, most sites name a sitemap and point to their own address. The second-largest group names none at all, and the groups that point wrongly are small but not rare.

**A third of Polish robots.txt files name no sitemap** (What the robots.txt of .pl sites says about sitemaps, by the worst pointer in the file · n = 6,834 sites with a robots.txt text file · 95% intervals)

| Item | Value | 95% interval | n |
| --- | --- | --- | --- |
| No Sitemap line | 35% | 34%–37% | 6,834 |
| Points to its own address | 59% | 58%–60% | 6,834 |
| Points only to other hosts | 1.7% | 1.4%–2% | 6,834 |
| Another protocol or www twin | 3.1% | 2.7%–3.5% | 6,834 |
| Unusable as written | 0.8% | 0.6%–1.1% | 6,834 |

Source: Outofplace census, 24 Sept 2026; Tranco list Y8YYG. Chart: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/charts/en/chart-a-third-of-polish-robots-txt-files-name-no-sitemap.png Data: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/2026-09-24/verdicts.csv

Size doesn't help. Among the sites in the sample's top 1,000, 59% name a sitemap, fewer than the 65% of the sites ranked below them. The gap goes with the mix of software. 74% of the top sites run nothing we recognise as a shop platform, an SEO plugin or WordPress, against 45% of the sites below them. Compared kind with kind, top sites and the rest name a sitemap about equally often: 52% against 55% where we recognised nothing, 76% against 74% where we did.

## Should the sitemap be in robots.txt?

It isn't required: a search engine can find pages through links, and Google and Bing accept a sitemap submitted in their consoles. But a console reaches one engine, and every other crawler (Bing for a site only in Google's console, the AI crawlers, SEO tools) has the robots.txt line or a guess.

The AI crawler study fetched one sitemap per site for the largest 1,000 .pl domains on 23 September: the first `Sitemap:` URL when robots.txt names one, else `/sitemap.xml`, the conventional default. The two show what a crawler finds with the line and without it.

**A named sitemap almost always answers; a guessed /sitemap.xml mostly doesn't** (Share of the largest .pl domains whose sitemap request returned an XML sitemap or sitemap index · named n = 511, not named n = 362 · 95% intervals)

| Item | Value | 95% interval | n |
| --- | --- | --- | --- |
| Named in robots.txt: the first Sitemap URL | 96% | 95%–98% | 511 |
| Not named: a guess at /sitemap.xml | 34% | 29%–39% | 362 |

Source: Outofplace AI crawler study crawl, 23 Sept 2026; Tranco list Y8YYG. Chart: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/charts/en/chart-a-named-sitemap-almost-always-answers-a-guessed-sitemap-xml-most.png Data: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/2026-09-24/fetch_top1000.csv

96% of the named URLs answered with a sitemap. Without the line, `/sitemap.xml` did for 34% of the sites; the rest mostly answered with an error, some with an HTML page. Those sites may well have a sitemap elsewhere, submitted to Google, but a crawler that only reads robots.txt can't know.

## Who writes the Sitemap line: the platform or nobody

A CMS or shop platform can serve robots.txt itself, and then whether the file names a sitemap is up to the platform's settings. [WordPress 5.5](https://make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/) (2020) made its own robots.txt "reference the sitemap index" at `/wp-sitemap.xml`; SEO plugins replace it with their own.

**Whether a site names its sitemap varies with its platform** (Share of .pl sites with a robots.txt that name a sitemap in it, by the shop platform, SEO plugin or CMS recognised on the homepage · groups under 30 sites fold into other platforms · n per group in the table below · 95% intervals)

| Item | Value | 95% interval | n |
| --- | --- | --- | --- |
| IdoSell | 100% | 97%–100% | 217 |
| Shopify | 98% | 94%–100% | 127 |
| Rank Math (WordPress) | 90% | 84%–94% | 135 |
| WooCommerce | 85% | 81%–88% | 381 |
| Headless shop | 84% | 71%–92% | 44 |
| All in One SEO (WordPress) | 83% | 73%–90% | 76 |
| Yoast SEO (WordPress) | 78% | 74%–81% | 662 |
| Custom shop | 74% | 68%–79% | 214 |
| Magento | 72% | 66%–78% | 201 |
| WordPress, no SEO plugin found | 72% | 68%–76% | 437 |
| PrestaShop | 62% | 58%–66% | 553 |
| Other platforms | 62% | 54%–69% | 173 |
| No platform recognised | 54% | 53%–56% | 3,351 |
| Shoper | 47% | 40%–53% | 227 |
| AtomStore | 39% | 25%–55% | 36 |

Source: Outofplace census, 24 Sept 2026. Chart: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/charts/en/chart-whether-a-site-names-its-sitemap-varies-with-its-platform.png Data: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/2026-09-24/software.csv

Hosted shop platforms sit at both ends. On IdoSell and Shopify nearly every shop names its sitemap. On Shoper and AtomStore about half of the shops in our sample or fewer name one, and the two can't be told apart (AtomStore is a small group of 36 shops). From robots.txt alone we can't tell whether that comes from a platform setting, a per-shop option or files the shops edit themselves. WordPress with an SEO plugin sits high, and the largest group, the sites where we recognised no platform at all, is close to the bottom with 54%.

File names point to platforms too. `sitemap.xml` is the most common, on 45% of the sites with a pointer to their own host, and some of the next names belong almost entirely to one kind of software, such as `1_index_sitemap.xml` to PrestaShop and `googlesitemap` to Shoper:

## How often is the sitemap pointer wrong?

We sorted every `Sitemap:` value by what it points at, against the address the homepage actually landed on. Pointers to the site's own address are the norm. Pointers to another host are allowed by both Google and sitemaps.org and say nothing wrong on their own: a sitemap on a CDN, a shop's subdomain, a sister site. The rest fall into two groups.

- **Mismatched**: the site's own host with `http` instead of `https` (or another port), or the `www` twin of the host the homepage lands on. These work only if the server redirects, and by the sitemaps.org rule a sitemap fetched from `http://` speaks for `http://` URLs.
- **Broken**: no usable address as written. A relative path such as `/sitemap.xml`, which Google's rule for a fully qualified URL excludes; a host with no `https://`; `localhost`, a `.test` domain or a `dev.` host left over from staging; or a template whose placeholder was never filled in (`{page_domain}/…`, `undefined/…`).

**Most wrong sitemap pointers name http or the wrong www variant** (Share of .pl sites naming a sitemap with at least one pointer of each kind · allowed pointers to other hosts in grey, mismatched in light warm, broken in deep warm · n = 4,411 · 95% intervals)

| Item | Value | 95% interval | n |
| --- | --- | --- | --- |
| Another site | 3% | 2.5%–3.5% | 4,411 |
| www instead of the bare domain, or back | 2.4% | 2%–2.9% | 4,411 |
| http instead of https (or another port) | 2.4% | 2%–2.9% | 4,411 |
| A relative path (/sitemap.xml) | 1% | 0.7%–1.3% | 4,411 |
| Another host of the same site | 0.7% | 0.5%–1% | 4,411 |
| localhost, .test or a staging host | 0.1% | 0.1%–0.3% | 4,411 |
| An unfilled template ({page_domain}) | 0.1% | 0%–0.2% | 4,411 |
| No http(s):// before the host | 0.1% | 0%–0.2% | 4,411 |

Source: Outofplace census, 24 Sept 2026. Chart: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/charts/en/chart-most-wrong-sitemap-pointers-name-http-or-the-wrong-www-variant.png Data: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/2026-09-24/pointers.csv

Together, 266 sites name at least one wrong address: 211 mismatched and 55 broken. That would matter less if the wrong line sat next to a right one, but it rarely does: 247 of them name no sitemap on their own address at all, and for 53 every pointer is broken.

Wrong pointers are more common where we recognised no software. Where a platform, plugin or WordPress was recognised, 4.1% of the sites naming a sitemap point wrongly; where none was, 8.8% do.

## Multiple sitemaps in robots.txt

Most files name one sitemap: 85% of the sites that name any. The longest file we found lists 492. Google sets no limit, so a long list isn't an error, but it is a sign the site lists every sitemap where one sitemap index would do.

**Most Polish sites name a single sitemap in robots.txt** (Number of Sitemap lines in the robots.txt of .pl sites that name at least one · n = 4,411 · 95% intervals)

| Item | Value | 95% interval | n |
| --- | --- | --- | --- |
| 1 line | 85% | 84%–86% | 4,411 |
| 2 lines | 9.1% | 8.2%–9.9% | 4,411 |
| 3–5 lines | 3.3% | 2.8%–3.9% | 4,411 |
| 6–20 lines | 2% | 1.6%–2.5% | 4,411 |
| 21 or more | 0.5% | 0.4%–0.8% | 4,411 |

Source: Outofplace census, 24 Sept 2026. Chart: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/charts/en/chart-most-polish-sites-name-a-single-sitemap-in-robots-txt.png Data: https://www.outofplace.space/data/blog/sitemaps-robots-txt-polish-web/2026-09-24/lines.csv

Repeats are rarer: 45 sites list the same URL more than once, 115 extra lines in all. They do no harm, and they suggest a file that grew by copy and paste.

## How to add a sitemap to robots.txt

Open `https://your-domain/robots.txt` in a browser, or from a terminal:

```sh
curl -s https://www.example.pl/robots.txt | grep -i '^sitemap'
```

Finding no line at all is common. If your platform writes the file, look for the setting there before editing anything by hand.

Open the address your homepage lands on and note its protocol and host: `https://example.pl` and `https://www.example.pl` are different hosts to a crawler. The sitemap is usually at `/sitemap.xml`, `/sitemap_index.xml` (SEO plugins such as Yoast SEO) or `/wp-sitemap.xml` (WordPress without a plugin); open it and check that it is XML, not your 404 page.

Add the line anywhere in the file, outside or inside a group: it applies to every crawler either way.

```txt title="robots.txt"
Sitemap: https://www.example.pl/sitemap_index.xml
```

Not `/sitemap.xml`, not `www.example.pl/sitemap.xml`, not `http://` on an `https` site.

Under the [sitemaps.org protocol](https://www.sitemaps.org/protocol.html#index), a sitemap holds up to 50,000 URLs and 50 MB uncompressed; past that, split it and list the parts in a sitemap index, which may list 50,000 sitemaps. Name the index in robots.txt. A sitemap on another host (a CDN, a subdomain) is fine when your robots.txt points to it.

Moving to `https`, adding or dropping `www`, or copying a staging site to production is when pointers go stale. Fetch robots.txt again after the move and open each `Sitemap:` URL.

## How we measured

The sites come from our census of 24 September 2026 by `OutofplaceResearchBot` (see [what our crawler fetches and how to block it](https://www.outofplace.space/bot)), which reads robots.txt first and fetches only the homepage: 8,023 of the 9,208 .pl domains in the Tranco list answered with HTML, and 7,785 distinct sites remained after merging domains that redirect to the same address. For each, we took the robots.txt of the address the homepage landed on and kept the 6,834 that served it as a text file.

We read every `Sitemap:` line with the RFC 9309 parser of our AI crawler study, which accepts the key in any case and strips comments, and sorted each value by comparing it with the homepage's final address: the same protocol and host; the same host with another protocol or port; the `www` twin; another host of the same registrable domain; another site; or one of the broken forms above. A site's verdict is its worst pointer. The platform behind each site is the shop platform found by the classifier of [our Core Web Vitals study of Polish shops](https://www.outofplace.space/blog/core-web-vitals-polish-ecommerce), else the SEO plugin or WordPress by the markers they leave in the HTML.

The fetch test comes from the AI crawler study's own crawl a day earlier: the first 1,000 .pl domains of the same list that answered, of which we kept those whose robots.txt is a text file and left out requests our crawler's own robots.txt rules skipped; one sitemap request each, counted as a sitemap when the answer's root element was `urlset` or `sitemapindex`.

We read robots.txt as the server sent it; we didn't fetch the census sites' sitemaps, so a correct pointer to an empty or outdated sitemap counts as correct, and a mismatched one may well redirect to the right file. A text sitemap (one URL per line, which sitemaps.org allows) counts as no sitemap in the fetch test. Sites may submit sitemaps in Google Search Console or Bing Webmaster Tools, which we can't see: a missing line doesn't mean Google doesn't know the sitemap. The platform groups depend on what the homepage reveals; a site behind a custom theme may be filed under no platform.

## Further reading

- [The sitemaps.org protocol](https://www.sitemaps.org/protocol.html): The XML sitemap format, its size limits, sitemap index files, and how robots.txt points to a sitemap on the same or another host.

- [How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt): Google's rules for robots.txt, including the sitemap field: a fully qualified URL, any host, any number of lines.

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html): The robots.txt standard, and why a Sitemap record is one crawlers may read but that never ends a group.

- [AI crawlers vs the Polish web: robots.txt on 1,000 .pl sites](https://www.outofplace.space/blog/ai-crawlers-polish-web): Our study of robots.txt on the largest .pl sites: which AI crawlers they block, and how.