AI crawlers vs the Polish web: robots.txt on 1,000 .pl sites
We read robots.txt on the 1,000 top .pl sites: one in five blocks AI crawlers like GPTBot, mostly for training, rarely AI search. How to split the two.
On 23 September 2026 our crawler, OutofplaceResearchBot, read the robots.txt of the 1,000 most popular .pl websites to see which AI crawlers they let in. 20.5% of them block at least one AI crawler. That is statistically the same as the global top 1,000, where the share is 22.7%.
The difference is in which crawlers they block. Polish sites shut out the bots that collect training data and mostly leave AI search alone. Only 5.4% block an AI search crawler, against 13.9% globally. For anyone who wants to show up in ChatGPT, Claude or Perplexity answers, that is the right way round. For some sites it is an accident: an old list copied once and never updated.
- 20.5%
- 15.1%
- 5.4%
- 9.2%
What are AI crawlers? Training, search and fetchers
"AI crawler" covers three different jobs, and the vendors now give each its own user agent. The distinction decides everything that follows:
- Training crawlers collect pages to train models: GPTBot (opens in a new tab) (OpenAI), ClaudeBot (opens in a new tab) (Anthropic), Google-Extended (opens in a new tab), Applebot-Extended (opens in a new tab), CCBot (opens in a new tab) (Common Crawl), Bytespider (ByteDance), meta-externalagent (Meta), Amazonbot (opens in a new tab).
- AI search crawlers index pages so an assistant can find and cite them: OAI-SearchBot, Claude-SearchBot, PerplexityBot (opens in a new tab), DuckAssistBot (opens in a new tab).
- User-triggered fetchers open a page because someone asked the assistant to: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, MistralAI-User (opens in a new tab).
Blocking a training crawler keeps your pages out of future models. Blocking a search crawler or a fetcher keeps them out of the answers, and out of the links in those answers.
Two of the training "crawlers" don't crawl. Google-Extended and Applebot-Extended are only tokens: Googlebot (opens in a new tab) and Applebot fetch the page, and the token tells Google and Apple whether they may train on it. Blocking them leaves you in Google and Apple search, and blocking Google-Extended doesn't turn off AI Overviews (opens in a new tab) either: those follow your Googlebot rules. The fetchers are the other special case. OpenAI, Perplexity and Meta say theirs may not follow robots.txt at all, because a person asked for the page; Anthropic says Claude-User does.
Poland blocks as often as the world, but not the same bots
Across all AI crawlers, the gap between the .pl sample and the global control is 2.2 percentage points, well inside the margin of error. Split by purpose, the picture changes. Training crawlers are blocked equally often. AI search crawlers and user-triggered fetchers are blocked less than half as often in Poland.
Poland blocks training crawlers as often as the world, AI search far less
Share of sites whose robots.txt blocks at least one crawler of each kind for the homepage · n = 1,000 per sample · 23 Sept 2026
- Top 1,000 .pl
- Global top 1,000
Show the data
| Item | Top 1,000 .pl | Global top 1,000 |
|---|---|---|
| Any AI crawler | 21% | 23% |
| Training crawlers | 21% | 22% |
| AI search crawlers | 5.4% | 14% |
| User-triggered fetchers | 5.3% | 13% |
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
GPTBot vs OAI-SearchBot: two separate switches
The clearest view is per vendor. Of the Polish sites that block GPTBot, 28% also block OAI-SearchBot. In the global sample it is 56%. For Anthropic the split is starker: 24% against 60%.
Most Polish sites that block training leave AI search open
Sites that block the vendor's training crawler (GPTBot, ClaudeBot), by whether they also block its search crawler (OAI-SearchBot, Claude-SearchBot)
- Also blocks search
- Blocks training only
Show the data
| Item | Also blocks search | Blocks training only | n |
|---|---|---|---|
| OpenAI · Poland | 28% | 72% | 151 |
| OpenAI · Global | 56% | 44% | 154 |
| Anthropic · Poland | 24% | 76% | 107 |
| Anthropic · Global | 60% | 40% | 152 |
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
Globally, a site that blocks AI crawlers usually blocks the search ones too. In Poland, the typical blocker names a handful of training crawlers and stops there: 11.2% of the Polish sample blocks training crawlers only, almost twice the global 5.9%.
Which AI crawlers are blocked most
GPTBot leads, at 15.1% of Polish sites and 15.4% globally. OpenAI announced it in August 2023, and most blocklists start with it. Common Crawl's CCBot, Amazonbot, ByteDance's Bytespider and ClaudeBot follow. The newest agents barely register: Claude-SearchBot is blocked by 2.7% of Polish sites, mostly because the lists were written before it existed.
GPTBot is the most blocked AI crawler on .pl sites
Share of the top 1,000 .pl domains whose robots.txt blocks each crawler for the homepage · 95% intervals · training crawlers highlighted
Block any AI crawler: 21%
Show the data
| Item | Value | 95% interval | n |
|---|---|---|---|
| GPTBot | 15% | 13%–17% | 1,000 |
| CCBot | 13% | 11%–16% | 1,000 |
| Amazonbot | 12% | 10%–14% | 1,000 |
| Bytespider | 12% | 9.7%–14% | 1,000 |
| ClaudeBot | 11% | 8.9%–13% | 1,000 |
| meta-externalagent | 11% | 8.8%–13% | 1,000 |
| Google-Extended | 8.4% | 6.8%–10% | 1,000 |
| Applebot-Extended | 8.4% | 6.8%–10% | 1,000 |
| ChatGPT-User | 4.7% | 3.6%–6.2% | 1,000 |
| PerplexityBot | 4.6% | 3.5%–6.1% | 1,000 |
| OAI-SearchBot | 4.4% | 3.3%–5.9% | 1,000 |
| meta-externalfetcher | 4% | 3%–5.4% | 1,000 |
| DuckAssistBot | 3.7% | 2.7%–5.1% | 1,000 |
| Perplexity-User | 3.5% | 2.5%–4.8% | 1,000 |
| Claude-SearchBot | 2.7% | 1.9%–3.9% | 1,000 |
| Claude-User | 2.3% | 1.5%–3.4% | 1,000 |
| MistralAI-User | 2.3% | 1.5%–3.4% | 1,000 |
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
Almost all of it is deliberate. A site that disallows everything under User-agent: * blocks every crawler without a group of its own, AI or not. That wildcard floor accounts for only 1.9% of Polish sites for GPTBot; the rest name the bot.
The dumbbell below puts each crawler's Polish and global shares side by side. The two samples agree on training crawlers. They part ways on everything that serves answers.
The gap opens on search crawlers and fetchers
Share of sites blocking each crawler for the homepage · top 1,000 .pl against the global top 1,000
- Top 1,000 .pl
- Global top 1,000
Show the data
| Item | Top 1,000 .pl | Global top 1,000 |
|---|---|---|
| GPTBot | 15% | 15% |
| CCBot | 13% | 18% |
| Amazonbot | 12% | 13% |
| Bytespider | 12% | 17% |
| ClaudeBot | 11% | 15% |
| meta-externalagent | 11% | 14% |
| Google-Extended | 8.4% | 14% |
| Applebot-Extended | 8.4% | 13% |
| ChatGPT-User | 4.7% | 11% |
| PerplexityBot | 4.6% | 13% |
| OAI-SearchBot | 4.4% | 8.9% |
| meta-externalfetcher | 4% | 9.9% |
| DuckAssistBot | 3.7% | 9.5% |
| Perplexity-User | 3.5% | 10% |
| Claude-SearchBot | 2.7% | 9.2% |
| Claude-User | 2.3% | 9.5% |
| MistralAI-User | 2.3% | 9.6% |
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
Bigger sites block more
Blocking falls with popularity. 33% of the 100 most popular .pl domains block at least one AI crawler, against 15.8% of those ranked 501 to 1,000. The slope is steeper than in the global sample, which goes from 27% to 21.2%. The largest Polish sites are publishers, marketplaces and portals: the businesses with the most text to protect and the lawyers to ask the question.
A third of the top 100 .pl sites block AI crawlers
Share blocking at least one AI crawler, by rank within each sample · n = 100, 400 and 500
- Top 1,000 .pl
- Global top 1,000
Show the data
| Item | Top 1,000 .pl | Global top 1,000 |
|---|---|---|
| Top 100 | 33% | 27% |
| 101–500 | 23% | 24% |
| 501–1,000 | 16% | 21% |
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
The size effect holds crawler by crawler. GPTBot is blocked by 23% of the top 100 and 11.6% of the sites ranked 501 to 1,000, and every training crawler follows the same slope. The search crawlers stay low at every size.
The biggest .pl sites block training crawlers most, and search crawlers little at any size
Share of .pl sites blocking each crawler for the homepage, by rank within the sample · n = 100, 400 and 500
| Item | Top 100 | 101–500 | 501–1,000 |
|---|---|---|---|
| GPTBot | 23% | 18% | 12% |
| CCBot | 21% | 16% | 9.6% |
| Amazonbot | 16% | 16% | 8.8% |
| Bytespider | 19% | 14% | 8% |
| ClaudeBot | 14% | 13% | 8.6% |
| meta-externalagent | 15% | 14% | 6.8% |
| Google-Extended | 12% | 11% | 5.8% |
| Applebot-Extended | 11% | 11% | 5.6% |
| OAI-SearchBot | 7% | 4.3% | 4% |
| PerplexityBot | 6% | 4.8% | 4.2% |
| Claude-SearchBot | 4% | 2.3% | 2.8% |
Show as an image
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
Public institutions block less, and not on purpose. 11.9% of the 42 gov.pl domains in the sample block an AI crawler, and 16.1% of the 31 edu.pl ones; both groups are small, so read those as directions. Most of those blocks aren't an AI policy at all. sejm.gov.pl, dziennikustaw.gov.pl and wroclaw.sa.gov.pl let in Googlebot and disallow everyone else (opens in a new tab), which shuts out every AI crawler along with Bing and DuckDuckGo. For the parliament and the journal of laws, that also means Copilot and ChatGPT search won't read the sources people ask them about.
Lists go stale
Blocklists are written once and copied for years. 7.2% of Polish sites still block anthropic-ai, a name Anthropic stopped using (its documentation now lists ClaudeBot, Claude-SearchBot and Claude-User), while only 2.7% block Claude-SearchBot. For 4.1% of the sample the list is stale in the way that matters: it blocks an old name and lets the crawler that replaced it through.
Anthropic, to its credit, told 404 Media in 2024 (opens in a new tab) that ClaudeBot honours rules written for its two retired names. That is goodwill, not the standard: under RFC 9309 (opens in a new tab) a rule for anthropic-ai doesn't apply to ClaudeBot, and the next vendor to rename a crawler may not be as generous.
Cloudflare's managed robots.txt and Content Signals
3.1% of Polish sites serve Cloudflare's managed robots.txt (opens in a new tab): one switch in the dashboard, and the CDN puts a block of rules in front of the site's own file and keeps it up to date. The block disallows exactly the eight training crawlers in our list, from GPTBot to Amazonbot, and no search crawler. Those sites, among them the Polska Press regional titles such as poranny.pl, nto.pl, pomorska.pl and gazetawroclawska.pl, make up 15.1% of all Polish blockers and 24% of the sites that block training only. They are part of why Poland leans that way, though not all of it: take them out and training-only blocks are still more common than in the global sample, where we found no managed file at all. Large international sites write their own.
The switch shows in the numbers. Behind Cloudflare, 25.6% of Polish sites block AI crawlers, against 18.2% elsewhere. Globally it is the other way round.
In Poland, sites behind Cloudflare block more
Share blocking at least one AI crawler, by whether the site is served through Cloudflare
- Behind Cloudflare
- Other hosting
Show the data
| Item | Behind Cloudflare | Other hosting |
|---|---|---|
| Poland | 26% | 18% |
| Global | 19% | 24% |
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
The managed file also carries a Content-Signal line (search=yes, ai-train=no), part of the Content Signals Policy (opens in a new tab) Cloudflare published in September 2025 for saying what content may be used for: search, AI input, AI training. 3.9% of Polish sites say ai-train=no this way, against 1.6% globally. If a robots.txt tester flags Content-Signal as an unknown directive, that is expected and harmless: under RFC 9309 a crawler skips lines it doesn't recognise. 79% of those lines come from the managed file, not from anyone typing them.
What is llms.txt, and who has one?
llms.txt (opens in a new tab) is a Markdown file that tells language models what a site is and where its important pages are, proposed by Jeremy Howard (opens in a new tab) in September 2024. The spec asks only for an H1 with the site's name; we also required at least one link, since a file without one points nowhere. 9.2% of the Polish sample serve such a file at /llms.txt. Another 10% answer /llms.txt with an HTML page and a 200 status, which is what a model reading it receives instead, and what a careless checker counts as "has llms.txt".
Hosting companies got there first: home.pl, nazwa.pl, cyberfolks.pl, dhosting.pl and kei.pl all serve a valid file, and so do mbank.pl, t-mobile.pl, rossmann.pl, mediamarkt.pl and, among public bodies, uokik.gov.pl. Google has said its Search ignores the file (opens in a new tab); it is there for the other assistants. We serve one ourselves: our llms.txt lists our studies, case studies and services, and llms-full.txt carries the full text of every post.
Most sites have no llms.txt; one in ten answer with a web page
What GET /llms.txt returned · n = 1,000 per sample · errors and paths our crawler was asked not to fetch count as none
- Valid llms.txt
- Wrong format
- HTML page (soft 404)
- None
Show the data
| Item | Valid llms.txt | Wrong format | HTML page (soft 404) | None | n |
|---|---|---|---|---|---|
| Top 1,000 .pl | 9.2% | 2.4% | 10% | 78% | 1,000 |
| Global top 1,000 | 14% | 3.4% | 9.6% | 74% | 1,000 |
The chart as an image, free to reuse under CC BY 4.0 with credit to Outofplace and a link to this page.
The rest of the machine-readable web
Crawlers read more than robots.txt. Here Polish sites are close to the world on the basics and behind on the details that help a machine place a page: hreflang alternates (opens in a new tab), and a sitemap listed where crawlers look for it (opens in a new tab). Zstandard compression (opens in a new tab) is several times more common on .pl homepages, but that is Cloudflare again: all but one of the Polish sites that sent it came through the CDN.
| Signal | Poland | Global |
|---|---|---|
| <html lang> | 89% | 89% |
| Open Graph tags | 71% | 70% |
| JSON-LD | 49% | 47% |
| Sitemap listed in robots.txt | 53% | 62% |
| hreflang | 18% | 35% |
| HTTP/2 | 82% | 89% |
| HTTP/3 advertised | 30% | 36% |
| Brotli | 31% | 32% |
| Zstandard | 8.3% | 2.4% |
Three policies worth copying
Some Polish sites have clearly thought about it. We checked each of these files by hand on the day of the crawl.
- Training no, answers yes. The Wirtualna Polska titles (wp.pl, o2.pl, money.pl, pudelek.pl, abczdrowie.pl, dobreprogramy.pl) disallow GPTBot, CCBot and Bytespider (opens in a new tab), and give OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User their own
Allow: /groups. A publisher saying exactly what it wants. - Everyone welcome, by name. mediamarkt.pl and euro.com.pl have
Allow: /groups (opens in a new tab) for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and more. For a retailer, an assistant that recommends a product is a shop window. - Let the CDN keep the list. The Polska Press regional titles use Cloudflare's managed file, which blocks the training crawlers and stays current without anyone editing it.
How to block AI crawlers in robots.txt
Should you block AI crawlers?
There is no single right policy. A publisher that sells its archive has different interests from a shop that wants to be recommended. But whichever you choose, choose by purpose, not by vendor:
Blocking every AI crawler
- Out of future models
- Out of ChatGPT, Claude and Perplexity answers
- No links from AI answers
- Lists need constant updating
Blocking training only
- Out of future models
- Cited and linked in AI answers
- Readers who ask an assistant still find you
- Search agents change less often
A robots.txt that blocks AI training and keeps AI search
This is the shape of a robots.txt that blocks training and keeps AI search. The training group is the same eight crawlers Cloudflare's managed file blocks. Our own site does the opposite and lets every crawler in, because being cited is the point of a studio blog.
# AI search and user-triggered fetches: they cite and link to you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Named groups don't inherit from *, so repeat private paths
Disallow: /cart/
Disallow: /account/
# Model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /cart/
Disallow: /account/
Sitemap: https://www.example.pl/sitemap.xmlDecide per purpose
Training, AI search, user-triggered fetches. Write down what you want for each before you touch the file.
Name the current tokens
Take them from the vendors' own pages, not from a list someone posted in 2023.
Repeat private paths in every named group
A crawler that matches a named group ignores
User-agent: *entirely.Test it
Paste the file into the checker below and look at the paths that matter to you.
Cover the crawlers robots.txt can't
Copilot rides on Bingbot, so its controls are the
noarchiveandnocachemeta tags (opens in a new tab). Google's AI Overviews follow Googlebot and snippet controls such asnosnippet(opens in a new tab). Fetchers that may ignore robots.txt need a CDN or server rule if they must be stopped.Check it every quarter
New agents appear a few times a year. A stale list is worse than none, because it looks like a decision.
Paste your robots.txt here to see what each AI crawler may fetch. It runs in your browser on the same parser we used for the study; nothing is sent anywhere.
Tool
Which AI crawlers can read your site?
Paste your robots.txt. It is checked right here in your browser with the same RFC 9309 parser and bot list as our study; nothing you paste is sent anywhere.
How we measured
Which sites count, and why 1,000
The sample is the first 1,000 .pl domains in the Tranco list that answered our requests. 23 of them disallow our crawler from their homepage in robots.txt; we still read their robots.txt (it is public by definition) but fetched nothing else. Domains that failed DNS, refused connections or returned server errors were skipped and replaced by the next one on the list. With 1,000 sites, the 95% interval around the headline share runs from 18.1% to 23.1%.
How a site's policy is classified
Each site gets one class, first match wins. Blanket block: Googlebot is blocked too, so the rule is not about AI. All AI: every training crawler and at least one search crawler or fetcher are blocked. Stale list: a name the vendor no longer documents is blocked, while the crawler that replaced it is not. Training only: training crawlers blocked, no search crawler or fetcher. Inconsistent: a search crawler or fetcher is blocked while a training crawler is let through. Open: no AI crawler blocked. The rules and their tests are in the repository.
What this can't tell you
robots.txt is only what a site asks for. Some sites block AI crawlers at the firewall or CDN instead, and some serve a bot challenge to every crawler; we counted challenges separately and never as "allowed". We evaluated the homepage path only: a site that blocks AI crawlers from its articles but not from / counts as open. A .pl domain is not the same as a Polish business, and popularity lists favour big sites. The numbers describe the most visited part of the .pl web on one day.
Should I block AI crawlers?
Decide by purpose. Block the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and others) if you don't want your pages in future models. Keep the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to be cited and linked in ChatGPT, Claude and Perplexity answers: blocking them takes you out of those answers.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, and treats the two settings independently. Blocking GPTBot keeps your pages out of training data; only blocking OAI-SearchBot keeps them out of ChatGPT search answers, and even then they can appear as plain navigational links.
Does blocking Google-Extended hurt my rankings or turn off AI Overviews?
Neither. Google says Google-Extended does not affect inclusion or ranking in Google Search; it controls whether your content is used to train Gemini models and to ground Gemini's answers. AI Overviews and AI Mode are part of Search and follow your Googlebot rules and snippet controls such as nosnippet instead.
How do I keep my site out of Microsoft Copilot?
Microsoft has no separate AI crawler: Bing and Copilot use the same Bingbot crawl. Microsoft documents two meta tags instead: noarchive keeps a page out of Copilot answers and out of training, and nocache limits Copilot to the URL, title and snippet. Both leave the page in Bing search.
How many Polish websites block AI crawlers?
In our September 2026 crawl, 20.5% of the 1,000 most popular .pl domains blocked at least one AI crawler in robots.txt (95% interval 18.1%–23.1%). GPTBot was blocked most often, by 15.1%.
Do I need llms.txt?
No, but it is cheap and harmless: a Markdown file, usually at /llms.txt, that tells language models what the site is and links to its key pages. Google says its Search ignores it, and no major AI company has said it uses the file for ranking, so treat it as a courtesy, not a lever. Only 9.2% of the top .pl sites serve a valid one.
Do AI crawlers respect robots.txt?
It stops the ones that choose to follow it. The major vendors say their training and search crawlers obey robots.txt, but OpenAI, Perplexity and Meta say their user-triggered fetchers may not, because a person asked for the page. robots.txt is a request, not an access control; enforcing a policy takes server or CDN rules.