Data
AI crawler access in robots.txt, 10,000 domains
In the 2026-09-03 run, 15.1% of the 8,036 top-10,000 Tranco domains whose robots.txt we could read blocked Common Crawl's CCBot at the site root, and 14.3% blocked OpenAI's GPTBot. 81.4% of the read domains never mention GPTBot at all. CDN-level blocks are not counted, so real blocking is higher.
7 days free · card required
15.1%
of the domains whose robots.txt we could read block Common Crawl's CCBot at the site root
n=8,036 domains · run · Tranco list K9QPW
Method- block OpenAI's GPTBot
- 14.3%
- n=8,036 domains
- block OpenAI's OAI-SearchBot
- 5.8%
- n=8,036 domains
- say nothing about GPTBot
- 81.4%
- n=8,036 domains
Last updated
Key facts
Read these shares as floors, not totals. Bot rules applied at a CDN never appear in robots.txt, so a site closed at the edge can read here as unspecified or allowed; why the blocked share is a floor is set out below.
So what this page measures is the share of the web that has written its AI-crawler policy down in the one file every operator has agreed to read. Real blocking is higher than these figures, and real access is lower. A crawler that is allowed here may still be refused at the edge, so check your own server logs before you trust either number.
What did we measure?
AI crawlers are how a page reaches an AI answer. Each operator runs separate agents for separate jobs: one collects training data, one builds the index its search answers draw on, and one fetches a page live when a user asks. A site owner's one widely honoured lever over all of them is robots.txt, and until someone counts, nobody knows how much of the web has pulled it, or for which job.
So we counted. For every domain in a single dated ranking of the web's most visited sites, we fetched /robots.txt, parsed its user-agent groups, and recorded what it says about each of the AI crawler user-agent tokens our product tracks. The result is three numbers per crawler: how many domains name it and permit it, how many domains block it at the site root, and how many have never mentioned it at all. Failed fetches are reported as failed fetches and are never folded into any of the three.
How often is each AI crawler blocked?
Shares are of the 8,036 domains whose robots.txt we successfully read. A crawler with fewer than 1,000 readings is reported as not measured, never as zero.
The table shows the AI crawlers of the major AI operators, plus Googlebot and bingbot, whose search indexes feed Google's AI Overviews and Microsoft Copilot. The CSV carries every crawler in the study, including the scrapers and the remaining training-only agents.
GPTBot · OpenAI
- Blocked
- 14.3%
- Allowed
- 4.3%
- Unspecified
- 81.4%
OAI-SearchBot · OpenAI
- Blocked
- 5.8%
- Allowed
- 4.2%
- Unspecified
- 90.0%
ChatGPT-User · OpenAI
- Blocked
- 7.1%
- Allowed
- 4.3%
- Unspecified
- 88.6%
ClaudeBot · Anthropic
- Blocked
- 13.4%
- Allowed
- 3.4%
- Unspecified
- 83.2%
Claude-SearchBot · Anthropic
- Blocked
- 5.5%
- Allowed
- 1.7%
- Unspecified
- 92.8%
Claude-User · Anthropic
- Blocked
- 5.5%
- Allowed
- 1.7%
- Unspecified
- 92.9%
PerplexityBot · Perplexity
- Blocked
- 8.2%
- Allowed
- 3.9%
- Unspecified
- 87.9%
Perplexity-User · Perplexity
- Blocked
- 5.7%
- Allowed
- 1.5%
- Unspecified
- 92.8%
Google-Extended · Google
- Blocked
- 12.3%
- Allowed
- 3.4%
- Unspecified
- 84.2%
Googlebot · Google
- Blocked
- 1.9%
- Allowed
- 6.9%
- Unspecified
- 91.2%
bingbot · Microsoft
- Blocked
- 2.2%
- Allowed
- 5.3%
- Unspecified
- 92.5%
Bytespider · ByteDance
- Blocked
- 14.3%
- Allowed
- 1.1%
- Unspecified
- 84.6%
CCBot · Common Crawl
- Blocked
- 15.1%
- Allowed
- 1.8%
- Unspecified
- 83.0%
Meta-ExternalAgent · Meta
- Blocked
- 12.4%
- Allowed
- 1.5%
- Unspecified
- 86.2%
| Crawler | Operator | Blocked | Allowed | Unspecified |
|---|---|---|---|---|
| GPTBot | OpenAI | 14.3% | 4.3% | 81.4% |
| OAI-SearchBot | OpenAI | 5.8% | 4.2% | 90.0% |
| ChatGPT-User | OpenAI | 7.1% | 4.3% | 88.6% |
| ClaudeBot | Anthropic | 13.4% | 3.4% | 83.2% |
| Claude-SearchBot | Anthropic | 5.5% | 1.7% | 92.8% |
| Claude-User | Anthropic | 5.5% | 1.7% | 92.9% |
| PerplexityBot | Perplexity | 8.2% | 3.9% | 87.9% |
| Perplexity-User | Perplexity | 5.7% | 1.5% | 92.8% |
| Google-Extended | 12.3% | 3.4% | 84.2% | |
| Googlebot | 1.9% | 6.9% | 91.2% | |
| bingbot | Microsoft | 2.2% | 5.3% | 92.5% |
| Bytespider | ByteDance | 14.3% | 1.1% | 84.6% |
| CCBot | Common Crawl | 15.1% | 1.8% | 83.0% |
| Meta-ExternalAgent | Meta | 12.4% | 1.5% | 86.2% |
What was measured, and how big was the sample?
What was measured, over what frame, at what size, stated so that the number carries its own denominator.
- robots.txt files read
- 8,036
- 95.3% of 8,432 sites
- excluded: no website answered
- 1,568
- of 10,000 in the frame
- sites whose robots.txt could not be read
- 396
- of 8,432 sites
- Sampling frame
- Tranco list K9QPW for 2026-09-01, top 10,000 ranked domains, downloaded 2026-09-03
- Domains in the frame
- 10,000
- Excluded: no website
- 1,568 answered no HTTP request on the apex, on www or over plain HTTP
- Study population
- 8,432 (84.3% of the frame)
- robots.txt read
- 8,036 (95.3% of the population)
- llms.txt found
- 966 of 7,781 probed (12.4%)
- Run date
- 2026-09-03
- Refresh cadence
- Quarterly. Every run keeps its own dated URL.
- Collector
robotscrawl/, announcing itself as1 AttensiraResearchBot/1. 0 ( + https:/ / attensira. com/ data/ ai-crawler-robots-access)
Can I download the data?
Aggregate rows, CC BY 4.0. Cite it as Attensira, "AI crawler access in robots.txt, 10,000 domains", run 2026-09-03.
Tranco publishes no redistribution licence for the combined list and one input (Cloudflare Radar) is CC BY-NC, so per-domain rows are not republished; the Tranco list ID and date published with this run are sufficient to download the identical frame and re-run the study.
Licence: CC BY 4.0. Quoted third-party material belongs to the source it is attributed to.
Do sites treat OpenAI's training crawler differently from its search crawler?
GPTBot collects training data; OAI-SearchBot builds the index OpenAI's search answers draw on. More sites block the first (14.3%) than the second (5.8%), and most say nothing about either.
GPTBottraining
- Blocked
- 14.3%
- Allowed
- 4.3%
- Unspecified
- 81.4%
OAI-SearchBotsearch index
- Blocked
- 5.8%
- Allowed
- 4.2%
- Unspecified
- 90.0%
- Sample
- 8,036 domains
- Read
- 3 Sep 2026
Which crawler should you block, and which keeps you in AI answers?
Block by job, not by brand. The operators say what each agent does. OpenAI's bot documentation says GPTBot crawls content that 'may be used in training', and that 'sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers'. Anthropic's crawler documentation says restricting ClaudeBot signals that a site's future materials should be excluded from training, while blocking Claude-SearchBot 'may reduce your site's visibility and accuracy in user search results'. Retrieved 2026-10-04 from each operator's published crawler documentation.
So a site that wants to stay out of training sets but stay in AI answers blocks the training agents and allows the search and user agents. The per-crawler figures on this page show how few sites have written that distinction down: for most crawlers, most of the top of the web says nothing at all, which means the default allow applies.
Writing the rule is half of it. robots.txt says what a site asks for, not what reaches it. Attensira's AI traffic report reads crawler fetches on the server side, because AI crawlers do not run JavaScript and a client-side beacon never sees them, so a workspace can check which agents actually fetched which pages after a robots.txt change. To draft the file itself, the free robots.txt generator writes per-crawler rules, and the crawler directory documents each agent this study measures.
How was the study run?
The sampling frame
The frame is the top 10,000 ranked domains of one specific Tranco list, identified on this page and in the CSV header by Tranco's permanent list ID and the date the list is for. Tranco is a research ranking built, per tranco-list.eu, from 'one million domains obtained by averaging all four rankings over the past 30 days', which makes it far more stable day to day than any single provider's list. That matters here, because a frame that churns would make run-over-run comparisons meaningless. We record the list ID rather than saying 'the top 10,000 domains' because the second phrase is not reproducible and the first one is: anyone can download the identical file from Tranco and re-run this study against it.
We do not republish the ranked domains themselves. Tranco states no redistribution licence for its combined list, and one of its four inputs, Cloudflare Radar, is CC BY-NC. Publishing the per-domain rows would be republishing someone else's dataset under our name, so the download on this page is the aggregate table only. The frame identifier is what reproducibility actually requires.
Fetching
A bounded pool of 32 workers fetches https://{domain}/robots.txt with a 10-second per-host timeout, following up to five redirects. A failing host gets exactly one retry after a short backoff. The collector identifies itself honestly as AttensiraResearchBot/1.0 with a link back to this page, and it respects each site's own robots.txt for the second request it makes: /llms.txt is an ordinary path, so we only probe it where the site permits our agent to read it. robots.txt itself is exempt from that check by definition, since it is the file that carries the instruction.
A 404 or 410 on /robots.txt is an authoritative answer, not a failure: it means the site has no robots.txt, and every crawler is therefore unspecified against it. Only a transport error, a timeout, a 5xx or a 429 counts as a failed fetch. Bodies that are not robots syntax at all (a single-page-app HTML shell, a JSON error document, a captive portal) are treated as no robots.txt rather than parsed, because parsing them would silently manufacture a real-looking 'unspecified' out of a broken response.
Parsing and what counts as blocked
The parser follows RFC 9309. Consecutive User-agent lines share the rules that follow them; field names are case-insensitive; text after a # is a comment; unknown fields such as Sitemap, Host and Crawl-delay are ignored rather than treated as errors; and rules appearing before any User-agent line belong to no group and are discarded, as the major crawlers discard them.
For each crawler token we resolve the group that applies to it (the group naming that token if one exists, otherwise the wildcard group) and evaluate the site root against that group's rules using RFC 9309 precedence: the longest matching pattern wins, and Allow wins a tie. The three verdicts follow from that. Allowed means a group names the crawler and permits the root. Blocked means the applicable group disallows the root. Unspecified means no group names the crawler and the default permits the root, which includes the very common case of a site with no robots.txt at all.
We evaluate the site root rather than a sample of deeper paths for two reasons. It is the only path that every domain in the frame has, so it is the only comparison that is like-for-like across 10,000 sites; and a site that disallows /admin has not made a decision about AI crawlers, so counting it as a block would inflate the headline with ordinary housekeeping.
Where the crawler list comes from
The user-agent tokens are the AI-purpose subset of the bot table our product uses to classify real crawler hits in customer server logs. Every token on this page carries a link to the operator's own published documentation for it, because a user-agent string with no operator source is an assertion rather than a fact, and the point of this page is to be the origin of a citable number rather than another site repeating one. Crawlers with no AI purpose, such as uptime monitors, link unfurlers and conventional SEO tools, are excluded; they would be honest noise here in exactly the way they are honest noise on a customer dashboard.
Googlebot and bingbot are the two search crawlers kept in the set, on purpose. The indexes they build are what Google's AI Overviews and Microsoft Copilot draw their web answers from, so a site that blocks either one leaves those AI answers as well as classic search. Google-Extended, the token Google offers for AI, covers Gemini training and grounding, not AI Overviews. That is why the chart shows both search crawlers beside the AI-only agents: their rows are the baseline that the AI-only rows are read against.
When this page refuses to render
The page is built from a committed, dated run and does not render at all if that run is missing, if it is older than two refresh cycles, or if we read too few of the study population's robots.txt files to stand behind the shares (a collection-rate floor every run must clear). A per-crawler row with fewer than 1,000 readings renders as 'not measured'. There is no degraded mode and no placeholder: a research page that cannot support its numbers should be absent, and absence is a great deal cheaper than a stale figure under a named byline.
How does Attensira report numbers?
The reporting rules this page is held to (an unmeasured value is never a zero, a rate carries its sample, surfaces are never averaged) are written down in one place and apply to every figure Attensira publishes.
Questions this page gets asked
Does blocking GPTBot keep my site out of ChatGPT answers?
How do I check which AI crawlers actually reach my site?
Does blocking a crawler in robots.txt actually stop it?
Why is 'unspecified' a separate number from 'allowed'?
What counts as blocked?
Which domains are in the sample, and can I get the list?
Why does a row sometimes say 'not measured' instead of a percentage?
How often does this update, and are old numbers kept?
Want whatever it takes to win the answer?
7 days free · card required