---
title: "AI crawlers vs the Polish web: robots.txt on 1,000 .pl sites"
description: "We read robots.txt on the 1,000 top .pl sites: one in five blocks AI crawlers like GPTBot, mostly for training, rarely AI search. How to split the two."
canonical_url: "https://www.outofplace.space/blog/ai-crawlers-polish-web"
language: "en"
last_updated: "2026-09-23"
alternates:
  pl: "https://www.outofplace.space/pl/blog/ai-crawlers-polish-web.md"
---

# AI crawlers vs the Polish web: robots.txt on 1,000 .pl sites

> We read robots.txt on the 1,000 top .pl sites: one in five blocks AI crawlers like GPTBot, mostly for training, rarely AI search. How to split the two.

Published 2026-09-23 · Category: Search & AI · Tags: AI crawlers, robots.txt, GPTBot, llms.txt, Poland

On 23 September 2026 our crawler, [OutofplaceResearchBot](https://www.outofplace.space/bot), read the robots.txt of the 1,000 most popular .pl websites to see which AI crawlers they let in. 20.5% of them block at least one AI crawler. That is statistically the same as the global top 1,000, where the share is 22.7%.

The difference is in _which_ crawlers they block. Polish sites shut out the bots that collect training data and mostly leave AI search alone. Only 5.4% block an AI search crawler, against 13.9% globally. For anyone who wants to show up in ChatGPT, Claude or Perplexity answers, that is the right way round. For some sites it is an accident: [an old list copied once and never updated](#lists-go-stale).

- One in five of the top 1,000 .pl sites blocks an AI crawler in robots.txt, as many as in the global top 1,000.
- GPTBot is the most blocked crawler in Poland and worldwide: 15.1% and 15.4%.
- Polish sites block training, not search. Among sites that block GPTBot, fewer than a third also block OAI-SearchBot; globally more than half do.
- Size matters: a third of the top 100 .pl sites block AI crawlers, against one in six ranked 501–1,000.
- One in seven Polish blockers didn't write their rules: they use [Cloudflare's managed robots.txt](#cloudflares-managed-robotstxt-and-content-signals). We found none in the global sample.

## What are AI crawlers? Training, search and fetchers

"AI crawler" covers three different jobs, and the vendors now give each its own user agent. The distinction decides everything that follows:

- **Training crawlers** collect pages to train models: [GPTBot](https://developers.openai.com/api/docs/bots) (OpenAI), [ClaudeBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic), [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers?hl=en), [Applebot-Extended](https://support.apple.com/en-us/119829), [CCBot](https://commoncrawl.org/ccbot) (Common Crawl), Bytespider (ByteDance), meta-externalagent (Meta), [Amazonbot](https://developer.amazon.com/amazonbot).
- **AI search crawlers** index pages so an assistant can find and cite them: OAI-SearchBot, Claude-SearchBot, [PerplexityBot](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), [DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot).
- **User-triggered fetchers** open a page because someone asked the assistant to: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, [MistralAI-User](https://docs.mistral.ai/robots).

Blocking a training crawler keeps your pages out of future models. Blocking a search crawler or a fetcher keeps them out of the answers, and out of the links in those answers.

Two of the training "crawlers" don't crawl. Google-Extended and Applebot-Extended are only tokens: [Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot?hl=en) and Applebot fetch the page, and the token tells Google and Apple whether they may train on it. Blocking them leaves you in Google and Apple search, and blocking Google-Extended doesn't turn off [AI Overviews](https://developers.google.com/search/docs/appearance/ai-features?hl=en) either: those follow your Googlebot rules. The fetchers are the other special case. OpenAI, Perplexity and Meta say theirs may not follow robots.txt at all, because a person asked for the page; Anthropic says Claude-User does.

## Poland blocks as often as the world, but not the same bots

Across all AI crawlers, the gap between the .pl sample and the global control is 2.2 percentage points, well inside the margin of error. Split by purpose, the picture changes. Training crawlers are blocked equally often. AI search crawlers and user-triggered fetchers are blocked less than half as often in Poland.

**Poland blocks training crawlers as often as the world, AI search far less** (Share of sites whose robots.txt blocks at least one crawler of each kind for the homepage · n = 1,000 per sample · 23 Sept 2026)

| Item | Top 1,000 .pl | Global top 1,000 |
| --- | --- | --- |
| Any AI crawler | 21% | 23% |
| Training crawlers | 21% | 22% |
| AI search crawlers | 5.4% | 14% |
| User-triggered fetchers | 5.3% | 13% |

Source: Outofplace crawl; Tranco list Y8YYG. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-poland-blocks-training-crawlers-as-often-as-the-world-ai-search.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/domains.csv

### GPTBot vs OAI-SearchBot: two separate switches

The clearest view is per vendor. Of the Polish sites that block GPTBot, 28% also block OAI-SearchBot. In the global sample it is 56%. For Anthropic the split is starker: 24% against 60%.

**Most Polish sites that block training leave AI search open** (Sites that block the vendor's training crawler (GPTBot, ClaudeBot), by whether they also block its search crawler (OAI-SearchBot, Claude-SearchBot))

| Item | Also blocks search | Blocks training only | n |
| --- | --- | --- | --- |
| OpenAI · Poland | 28% | 72% | 151 |
| OpenAI · Global | 56% | 44% | 154 |
| Anthropic · Poland | 24% | 76% | 107 |
| Anthropic · Global | 60% | 40% | 152 |

Source: Outofplace crawl, 23 Sept 2026. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-most-polish-sites-that-block-training-leave-ai-search-open.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/bot_policies.csv

Globally, a site that blocks AI crawlers usually blocks the search ones too. In Poland, the typical blocker names a handful of training crawlers and stops there: 11.2% of the Polish sample blocks training crawlers only, almost twice the global 5.9%.

## Which AI crawlers are blocked most

GPTBot leads, at 15.1% of Polish sites and 15.4% globally. OpenAI announced it in August 2023, and most blocklists start with it. Common Crawl's CCBot, Amazonbot, ByteDance's Bytespider and ClaudeBot follow. The newest agents barely register: Claude-SearchBot is blocked by 2.7% of Polish sites, mostly because the lists were written before it existed.

**GPTBot is the most blocked AI crawler on .pl sites** (Share of the top 1,000 .pl domains whose robots.txt blocks each crawler for the homepage · 95% intervals · training crawlers highlighted)

| Item | Value | 95% interval | n |
| --- | --- | --- | --- |
| GPTBot | 15% | 13%–17% | 1,000 |
| CCBot | 13% | 11%–16% | 1,000 |
| Amazonbot | 12% | 10%–14% | 1,000 |
| Bytespider | 12% | 9.7%–14% | 1,000 |
| ClaudeBot | 11% | 8.9%–13% | 1,000 |
| meta-externalagent | 11% | 8.8%–13% | 1,000 |
| Google-Extended | 8.4% | 6.8%–10% | 1,000 |
| Applebot-Extended | 8.4% | 6.8%–10% | 1,000 |
| ChatGPT-User | 4.7% | 3.6%–6.2% | 1,000 |
| PerplexityBot | 4.6% | 3.5%–6.1% | 1,000 |
| OAI-SearchBot | 4.4% | 3.3%–5.9% | 1,000 |
| meta-externalfetcher | 4% | 3%–5.4% | 1,000 |
| DuckAssistBot | 3.7% | 2.7%–5.1% | 1,000 |
| Perplexity-User | 3.5% | 2.5%–4.8% | 1,000 |
| Claude-SearchBot | 2.7% | 1.9%–3.9% | 1,000 |
| Claude-User | 2.3% | 1.5%–3.4% | 1,000 |
| MistralAI-User | 2.3% | 1.5%–3.4% | 1,000 |

Source: Outofplace crawl, 23 Sept 2026. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-gptbot-is-the-most-blocked-ai-crawler-on-pl-sites.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/summary_by_bot.csv

Almost all of it is deliberate. A site that disallows everything under `User-agent: *` blocks every crawler without a group of its own, AI or not. That wildcard floor accounts for only 1.9% of Polish sites for GPTBot; the rest name the bot.

The dumbbell below puts each crawler's Polish and global shares side by side. The two samples agree on training crawlers. They part ways on everything that serves answers.

**The gap opens on search crawlers and fetchers** (Share of sites blocking each crawler for the homepage · top 1,000 .pl against the global top 1,000)

| Item | Top 1,000 .pl | Global top 1,000 |
| --- | --- | --- |
| GPTBot | 15% | 15% |
| CCBot | 13% | 18% |
| Amazonbot | 12% | 13% |
| Bytespider | 12% | 17% |
| ClaudeBot | 11% | 15% |
| meta-externalagent | 11% | 14% |
| Google-Extended | 8.4% | 14% |
| Applebot-Extended | 8.4% | 13% |
| ChatGPT-User | 4.7% | 11% |
| PerplexityBot | 4.6% | 13% |
| OAI-SearchBot | 4.4% | 8.9% |
| meta-externalfetcher | 4% | 9.9% |
| DuckAssistBot | 3.7% | 9.5% |
| Perplexity-User | 3.5% | 10% |
| Claude-SearchBot | 2.7% | 9.2% |
| Claude-User | 2.3% | 9.5% |
| MistralAI-User | 2.3% | 9.6% |

Source: Outofplace crawl, 23 Sept 2026. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-the-gap-opens-on-search-crawlers-and-fetchers.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/summary_by_bot.csv

## Bigger sites block more

Blocking falls with popularity. 33% of the 100 most popular .pl domains block at least one AI crawler, against 15.8% of those ranked 501 to 1,000. The slope is steeper than in the global sample, which goes from 27% to 21.2%. The largest Polish sites are publishers, marketplaces and portals: the businesses with the most text to protect and the lawyers to ask the question.

**A third of the top 100 .pl sites block AI crawlers** (Share blocking at least one AI crawler, by rank within each sample · n = 100, 400 and 500)

| Item | Top 1,000 .pl | Global top 1,000 |
| --- | --- | --- |
| Top 100 | 33% | 27% |
| 101–500 | 23% | 24% |
| 501–1,000 | 16% | 21% |

Source: Outofplace crawl, 23 Sept 2026. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-a-third-of-the-top-100-pl-sites-block-ai-crawlers.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/domains.csv

The size effect holds crawler by crawler. GPTBot is blocked by 23% of the top 100 and 11.6% of the sites ranked 501 to 1,000, and every training crawler follows the same slope. The search crawlers stay low at every size.

**The biggest .pl sites block training crawlers most, and search crawlers little at any size** (Share of .pl sites blocking each crawler for the homepage, by rank within the sample · n = 100, 400 and 500)

| Item | Top 100 | 101–500 | 501–1,000 |
| --- | --- | --- | --- |
| GPTBot | 23% | 18% | 12% |
| CCBot | 21% | 16% | 9.6% |
| Amazonbot | 16% | 16% | 8.8% |
| Bytespider | 19% | 14% | 8% |
| ClaudeBot | 14% | 13% | 8.6% |
| meta-externalagent | 15% | 14% | 6.8% |
| Google-Extended | 12% | 11% | 5.8% |
| Applebot-Extended | 11% | 11% | 5.6% |
| OAI-SearchBot | 7% | 4.3% | 4% |
| PerplexityBot | 6% | 4.8% | 4.2% |
| Claude-SearchBot | 4% | 2.3% | 2.8% |

Source: Outofplace crawl, 23 Sept 2026. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-the-biggest-pl-sites-block-training-crawlers-most-and-search-cra.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/bot_policies.csv

Public institutions block less, and not on purpose. 11.9% of the 42 gov.pl domains in the sample block an AI crawler, and 16.1% of the 31 edu.pl ones; both groups are small, so read those as directions. Most of those blocks aren't an AI policy at all. sejm.gov.pl, dziennikustaw.gov.pl and wroclaw.sa.gov.pl [let in Googlebot and disallow everyone else](https://www.sejm.gov.pl/robots.txt), which shuts out every AI crawler along with Bing and DuckDuckGo. For the parliament and the journal of laws, that also means Copilot and ChatGPT search won't read the sources people ask them about.

## Lists go stale

Blocklists are written once and copied for years. 7.2% of Polish sites still block `anthropic-ai`, a name Anthropic stopped using (its documentation now lists ClaudeBot, Claude-SearchBot and Claude-User), while only 2.7% block Claude-SearchBot. For 4.1% of the sample the list is stale in the way that matters: it blocks an old name and lets the crawler that replaced it through.

Anthropic, to its credit, [told 404 Media in 2024](https://www.404media.co/websites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones/) that ClaudeBot honours rules written for its two retired names. That is goodwill, not the standard: under [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) a rule for `anthropic-ai` doesn't apply to ClaudeBot, and the next vendor to rename a crawler may not be as generous.

Well-behaved crawlers read it and obey it; nothing forces them to. Named groups also don't inherit anything from `User-agent: *`: a crawler follows the one group that names it and ignores the rest. If you give AI search crawlers their own `Allow` group, repeat your private paths in it.

## Cloudflare's managed robots.txt and Content Signals

3.1% of Polish sites serve [Cloudflare's managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/): one switch in the dashboard, and the CDN puts a block of rules in front of the site's own file and keeps it up to date. The block disallows exactly the eight training crawlers in our list, from GPTBot to Amazonbot, and no search crawler. Those sites, among them the Polska Press regional titles such as poranny.pl, nto.pl, pomorska.pl and gazetawroclawska.pl, make up 15.1% of all Polish blockers and 24% of the sites that block training only. They are part of why Poland leans that way, though not all of it: take them out and training-only blocks are still more common than in the global sample, where we found no managed file at all. Large international sites write their own.

The switch shows in the numbers. Behind Cloudflare, 25.6% of Polish sites block AI crawlers, against 18.2% elsewhere. Globally it is the other way round.

**In Poland, sites behind Cloudflare block more** (Share blocking at least one AI crawler, by whether the site is served through Cloudflare)

| Item | Behind Cloudflare | Other hosting |
| --- | --- | --- |
| Poland | 26% | 18% |
| Global | 19% | 24% |

Source: Outofplace crawl, 23 Sept 2026. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-in-poland-sites-behind-cloudflare-block-more.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/domains.csv

The managed file also carries a `Content-Signal` line (`search=yes, ai-train=no`), part of the [Content Signals Policy](https://contentsignals.org/) Cloudflare published in September 2025 for saying what content may be used for: search, AI input, AI training. 3.9% of Polish sites say `ai-train=no` this way, against 1.6% globally. If a robots.txt tester flags `Content-Signal` as an unknown directive, that is expected and harmless: under RFC 9309 a crawler skips lines it doesn't recognise. 79% of those lines come from the managed file, not from anyone typing them.

## What is llms.txt, and who has one?

[`llms.txt`](https://llmstxt.org) is a Markdown file that tells language models what a site is and where its important pages are, [proposed by Jeremy Howard](https://www.answer.ai/posts/2024-09-03-llmstxt.html) in September 2024. The spec asks only for an H1 with the site's name; we also required at least one link, since a file without one points nowhere. 9.2% of the Polish sample serve such a file at `/llms.txt`. Another 10% answer `/llms.txt` with an HTML page and a 200 status, which is what a model reading it receives instead, and what a careless checker counts as "has llms.txt".

Hosting companies got there first: home.pl, nazwa.pl, cyberfolks.pl, dhosting.pl and kei.pl all serve a valid file, and so do mbank.pl, t-mobile.pl, rossmann.pl, mediamarkt.pl and, among public bodies, uokik.gov.pl. Google has said [its Search ignores the file](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide?hl=en); it is there for the other assistants. We serve one ourselves: [our llms.txt](https://www.outofplace.space/llms.txt) lists our studies, case studies and services, and [llms-full.txt](https://www.outofplace.space/llms-full.txt) carries the full text of every post.

**Most sites have no llms.txt; one in ten answer with a web page** (What GET /llms.txt returned · n = 1,000 per sample · errors and paths our crawler was asked not to fetch count as none)

| Item | Valid llms.txt | Wrong format | HTML page (soft 404) | None | n |
| --- | --- | --- | --- | --- | --- |
| Top 1,000 .pl | 9.2% | 2.4% | 10% | 78% | 1,000 |
| Global top 1,000 | 14% | 3.4% | 9.6% | 74% | 1,000 |

Source: Outofplace crawl, 23 Sept 2026. Chart: https://www.outofplace.space/data/blog/ai-crawlers-polish-web/charts/en/chart-most-sites-have-no-llms-txt-one-in-ten-answer-with-a-web-page.png Data: https://www.outofplace.space/data/blog/ai-bots-polish-web/2026-09-23/domains.csv

## The rest of the machine-readable web

Crawlers read more than robots.txt. Here Polish sites are close to the world on the basics and behind on the details that help a machine place a page: [`hreflang` alternates](https://developers.google.com/search/docs/specialty/international/localized-versions?hl=en), and [a sitemap listed where crawlers look for it](https://www.sitemaps.org/protocol.html#submit_robots). [Zstandard compression](https://www.rfc-editor.org/rfc/rfc8878.html) is several times more common on .pl homepages, but that is Cloudflare again: all but one of the Polish sites that sent it came through the CDN.

## Three policies worth copying

Some Polish sites have clearly thought about it. We checked each of these files by hand on the day of the crawl.

- **Training no, answers yes.** The Wirtualna Polska titles (wp.pl, o2.pl, money.pl, pudelek.pl, abczdrowie.pl, dobreprogramy.pl) [disallow GPTBot, CCBot and Bytespider](https://www.wp.pl/robots.txt), and give OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User their own `Allow: /` groups. A publisher saying exactly what it wants.
- **Everyone welcome, by name.** mediamarkt.pl and euro.com.pl have [`Allow: /` groups](https://mediamarkt.pl/robots.txt) for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and more. For a retailer, an assistant that recommends a product is a shop window.
- **Let the CDN keep the list.** The Polska Press regional titles use Cloudflare's managed file, which blocks the training crawlers and stays current without anyone editing it.

## How to block AI crawlers in robots.txt

### Should you block AI crawlers?

There is no single right policy. A publisher that sells its archive has different interests from a shop that wants to be recommended. But whichever you choose, choose by purpose, not by vendor:

### A robots.txt that blocks AI training and keeps AI search

This is the shape of a robots.txt that blocks training and keeps AI search. The training group is the same eight crawlers Cloudflare's managed file blocks. Our own site does the opposite and [lets every crawler in](https://www.outofplace.space/robots.txt), because being cited is the point of a studio blog.

```txt title="robots.txt"
# AI search and user-triggered fetches: they cite and link to you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Named groups don't inherit from *, so repeat private paths
Disallow: /cart/
Disallow: /account/

# Model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.pl/sitemap.xml
```

Training, AI search, user-triggered fetches. Write down what you want for each before you touch the file.

Take them from [the vendors' own pages](#further-reading), not from a list someone posted in 2023.

A crawler that matches a named group ignores `User-agent: *` entirely.

Paste the file into the checker below and look at the paths that matter to you.

Copilot rides on Bingbot, so its controls are the [`noarchive` and `nocache` meta tags](https://www.bing.com/webmasters/help/robots-meta-tags-and-attributes-that-bing-supports-5198d240). Google's AI Overviews follow Googlebot and [snippet controls such as `nosnippet`](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag?hl=en). Fetchers that may ignore robots.txt need a CDN or server rule if they must be stopped.

New agents appear a few times a year. A stale list is worse than none, because it looks like a decision.

Paste your robots.txt here to see what each AI crawler may fetch. It runs in your browser on the same parser we used for the study; nothing is sent anywhere.

## How we measured

We took the [Tranco list Y8YYG](https://tranco-list.eu/list/Y8YYG/full) (created 22 September 2026, 30-day window, five providers) and walked it from the top, keeping the first 1,000 registrable .pl domains whose server answered. The global control is the first 1,000 domains of the same list, processed the same way. On 23 September our crawler fetched each domain's robots.txt and homepage, plus `/llms.txt`, `/llms-full.txt` and the sitemap, identifying itself as `OutofplaceResearchBot` with a link to a page explaining the study.

Each robots.txt was parsed under RFC 9309 and evaluated for the homepage path `/` for 38 user agents: 36 AI crawlers of 17 operators, retired names included, plus Googlebot and Bingbot for reference. The charts show the tokens each operator currently documents. A crawler counts as blocked when the rule that applies to it disallows `/`, whether it is named or falls back to `*`. Shares have 95% Wilson intervals; differences between the samples, Newcombe intervals.

The sample is the first 1,000 .pl domains in the Tranco list that answered our requests. 23 of them disallow our crawler from their homepage in robots.txt; we still read their robots.txt (it is public by definition) but fetched nothing else. Domains that failed DNS, refused connections or returned server errors were skipped and replaced by the next one on the list. With 1,000 sites, the 95% interval around the headline share runs from 18.1% to 23.1%.

Each site gets one class, first match wins. Blanket block: Googlebot is blocked too, so the rule is not about AI. All AI: every training crawler and at least one search crawler or fetcher are blocked. Stale list: a name the vendor no longer documents is blocked, while the crawler that replaced it is not. Training only: training crawlers blocked, no search crawler or fetcher. Inconsistent: a search crawler or fetcher is blocked while a training crawler is let through. Open: no AI crawler blocked. The rules and their tests are in the repository.

robots.txt is only what a site asks for. Some sites block AI crawlers at the firewall or CDN instead, and some serve a bot challenge to every crawler; we counted challenges separately and never as "allowed". We evaluated the homepage path only: a site that blocks AI crawlers from its articles but not from `/` counts as open. A .pl domain is not the same as a Polish business, and popularity lists favour big sites. The numbers describe the most visited part of the .pl web on one day.

## Further reading

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html): The standard for robots.txt: groups, matching and what a crawler must do when the file is missing or unreachable.

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots): GPTBot, OAI-SearchBot and ChatGPT-User, and what each is used for.

- [Anthropic: How site owners can block the crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler): ClaudeBot, Claude-SearchBot and Claude-User.

- [Google: Common crawlers, including Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers?hl=en): What Google-Extended controls, and what it doesn't.

- [Cloudflare: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/): What the one-click file blocks, and the Content-Signal line it adds.

- [llms.txt: the proposal and its format](https://llmstxt.org): The original specification: where the file lives and how it is structured.

- [OutofplaceResearchBot: what it fetches and how to block it](https://www.outofplace.space/bot): Who runs our crawler, how it identifies itself and how politely it crawls.