# CCBot vs GPTBot: the open web archive against one lab's training crawler

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> CCBot fills Common Crawl's open archive; GPTBot collects training data for OpenAI alone. What each block reaches, how to verify both, and what our logs saw.

- Canonical page: https://attensira.com/compare/ccbot-vs-gptbot
- Family: compare
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=ccbot-vs-gptbot

**In short:** CCBot crawls for Common Crawl, a non-profit that publishes an open archive of the web anyone can download, including model builders. GPTBot crawls for OpenAI's own model training. Blocking GPTBot keeps future pages from one lab. Blocking CCBot keeps them out of a public archive many parties reuse. Neither affects ChatGPT search.

**Readings, 3 Sep 2026 to 4 Oct 2026**

| Measure | Value |
| --- | --- |
| disallow CCBot at the root (1,215), most of any crawler read (n=8,036 domains) | 15.1% |
| disallow GPTBot at the root (1,147) (n=8,036 domains) | 14.3% |
| CCBot requests to attensira.com (n=143,601 AI-crawler requests) | 0 |
| of OpenAI requests were GPTBot's, about 830 (n=7,536 OpenAI requests) | ~11% |

_robots.txt of the Tranco top-10,000 (list K9QPW), read 2026-09-03; attensira.com AI traffic log, 5 Sep-4 Oct 2026, read over the Attensira MCP 2026-10-04, by claimed user agent · 4 Oct 2026_

## Verdict

Both are training-side crawlers, but they feed different things. GPTBot feeds one company. CCBot feeds a public archive that anyone, including many model builders, can download. Blocking CCBot is the broader decision; blocking GPTBot is the narrower one.

## The readings on this page

### The readings at the top

robots.txt of the Tranco top-10,000 (list K9QPW), read 2026-09-03; attensira.com AI traffic log, 5 Sep-4 Oct 2026, read over the Attensira MCP 2026-10-04, by claimed user agent · 3 Sep 2026 to 4 Oct 2026

| Reading | Value | Sample |
| --- | --- | --- |
| disallow CCBot at the root (1,215), most of any crawler read | 15.1% | n=8,036 domains |
| disallow GPTBot at the root (1,147) | 14.3% | n=8,036 domains |
| CCBot requests to attensira.com | 0 | n=143,601 AI-crawler requests |
| of OpenAI requests were GPTBot's, about 830 | ~11% | n=7,536 OpenAI requests |

### What does Perplexity say GPTBot is?

Asked "what is GPTBot" on 4 October 2026, Perplexity described it as OpenAI's training crawler and said nothing about where else the same pages can come from. That gap is the reason this pair exists. One stored answer, quoted whole.

Stored Perplexity answer to “what is GPTBot”, asked 4 Oct 2026 (n=1 answer, attensira.com workspace, Attensira MCP).

> GPTBot is OpenAI’s web crawler that scans publicly accessible websites to collect data that may be used to train and improve OpenAI’s language models (e.g., ChatGPT/GPT). It follows robots.txt rules, and site owners can block it if they don’t want their content used for training.

Named, in order: OpenAI. Attensira was not named.

### Which of the two actually reached our pages?

Of the 30 attensira.com pages AI crawlers fetched most in the 30 days to 4 October 2026, every one was fetched by an agent claiming to be GPTBot. None was fetched by CCBot. The crawler sites block most was not the one reading us.

attensira.com AI traffic log, page rows of the 30 most-fetched pages, Attensira MCP, attensira.com workspace · 5 Sep 2026 to 4 Oct 2026 · labelled by the user agent each request claimed · n=8 of the 30 most-fetched pages

| Our page | CCBot | GPTBot |
| --- | --- | --- |
| /contact-sales | Not seen | Fetched |
| / | Not seen | Fetched |
| /pricing | Not seen | Fetched |
| /about | Not seen | Fetched |
| /features/prompt-tracking | Not seen | Fetched |
| /features/content-optimization | Not seen | Fetched |
| /features/crawler-logs | Not seen | Fetched |
| /mcp | Not seen | Fetched |

### Which other training crawlers do large sites block?

robots.txt of the Tranco top-10,000 (list K9QPW of 2026-09-01), read by Attensira on 2026-09-03 · 3 Sep 2026 to 3 Sep 2026

| Reading | Value | Sample |
| --- | --- | --- |
| disallow Bytespider at the root (1,150 domains) | 14.3% | n=8,036 domains |
| disallow Meta-ExternalAgent at the root (994 domains) | 12.4% | n=8,036 domains |

## How do CCBot and GPTBot compare?

Each cell cites a claim in the Sources section below.

| Criterion | CCBot | GPTBot |
| --- | --- | --- |
| **Who gets the data** |  |  |
| Who receives the crawled content | A public archive anyone can download | One company's model training |
| Stated mission | Democratising access to web information | Training generative foundation models |
| **Identifying and blocking it** |  |  |
| robots.txt user-agent token | CCBot | GPTBot |
| Published user-agent string | CCBot/2.0 (https://commoncrawl.org/faq/) | ...compatible; GPTBot/1.4; +https://openai.com/gptbot |
| Machine-readable address list for verification | index.commoncrawl.org/ccbot.json, plus reverse DNS | openai.com/gptbot.json |
| Documented robots.txt control | Yes · Yes, with a published block directive | Yes · Yes, via its own token |
| **What we measured** |  |  |
| Top-10k domains disallowing it at the root (study of 3 Sep 2026, n=8,036) | 1,215 (15.1%) | 1,147 (14.3%) |
| Top-10k domains allowing it by name (n=8,036) | 149 (1.9%) | 349 (4.3%) |
| On attensira.com, 30 days to 4 Oct 2026 | None, of 143,601 AI-crawler requests | Seen on all 30 of our most-fetched pages |

## What is CCBot?

Common Crawl's crawler. It fills an open repository of web crawl data that Common Crawl describes as universally accessible, which downstream parties including model builders can download.

## What is GPTBot?

OpenAI's training crawler. It collects content that may be used to train OpenAI's generative foundation models, and it feeds one company rather than a public archive.

## When should you choose CCBot?

Block CCBot when your objection is to your pages being redistributed in an open dataset that any party can download and reuse, including for model training. It is the broadest single training-side block available in robots.txt.

## When should you choose GPTBot?

Block GPTBot when your objection is specifically to OpenAI training on your pages. It is narrower than blocking CCBot, has no documented effect on ChatGPT search, and leaves your pages available to the open archive.

## When is neither the right choice?

Neither block reaches copies that already exist: pages in past Common Crawl archives stay there, and content already collected by GPTBot is not described as removed. Neither affects whether ChatGPT search or Perplexity can read your pages today.

## Where does Attensira fit?

Whatever it takes to win the answer. On Growth and Business, Attensira counts crawler fetches at your edge next to what ChatGPT, Perplexity, Claude, Gemini and Google AI answer and cite.

## What sets CCBot and GPTBot apart?

- A CCBot block reaches every party that downloads the open archive, not one company, which is why it is the broadest single training-side line you can write.
- Common Crawl documents reverse DNS verification as well as an address list, so a CCBot claim can be checked two ways; GPTBot is checked against openai.com/gptbot.json only.
- Neither block removes anything already collected, and neither changes whether ChatGPT search or Perplexity can cite you today.

## What is the difference between CCBot and GPTBot?

Who receives what they collect. GPTBot is OpenAI's crawler, and OpenAI says it is used to crawl content that may be used in training its generative AI foundation models. What GPTBot collects goes to one company. CCBot is Common Crawl's crawler. Common Crawl is a non-profit whose stated aim is to democratise access to web information by maintaining an open repository of web crawl data. What CCBot collects goes into an archive that anyone can download.

That difference decides what a block achieves. A GPTBot block is a message to one lab. A CCBot block keeps your future pages out of a dataset that many parties use, including researchers and model builders. Neither block is about search visibility; both are about reuse.

## Why does CCBot appear on lists of AI crawlers if it does not train models?

Because its output is one of the most widely used sources of web text. Common Crawl publishes its data in formats designed for reuse and encourages research on it. Model builders are among the users. So CCBot sits on the training side of a robots.txt policy even though Common Crawl itself trains nothing: it is the upstream of many downstream uses you cannot see or list.

That makes CCBot the broadest single training-side decision you can write in robots.txt. A GPTBot rule covers one named crawler from one named lab. A CCBot rule covers every party that would otherwise have got your future pages from the open archive.

## What does each block reach, and what does it miss?

A GPTBot block reaches GPTBot. OpenAI's crawler page does not describe the company's other data sources, so a GPTBot rule does not, by itself, tell you whether OpenAI could obtain the same pages another way. A CCBot block reaches future Common Crawl archives. It does not reach past archives, which remain available, and it does not reach anyone who crawls the web with their own crawler.

Neither block touches search. ChatGPT search depends on OAI-SearchBot. Google Search depends on Googlebot. Perplexity depends on PerplexityBot. If you write CCBot and GPTBot rules and nothing else, every answer engine that searches the web can still read, cite and link your pages.

Answer engines describe GPTBot the same narrow way its operator does. Ask one what GPTBot is and you get OpenAI's training crawler, robots.txt control and the opt-out. You do not get the other route by which the same page can reach a training set, which is the one CCBot opens.

## What should your robots.txt say?

For a site that wants to stay out of open datasets and OpenAI's training while staying visible in search and answers:

```
User-agent: CCBot
Disallow: /

User-agent: GPTBot
Disallow: /
```

If you object only to one lab training on your pages and are content to be in the open archive, block GPTBot alone. If you object to reuse in general, CCBot is the line that does the most work, and you will probably want the other labs' training crawlers in the same group. Keep OAI-SearchBot, PerplexityBot and Googlebot out of that group, or give them their own allow groups, so a reuse decision does not quietly become a visibility decision.

## How do you verify each crawler?

CCBot identifies itself as `CCBot/2.0` with the URL commoncrawl.org/faq. Common Crawl publishes its IP ranges as JSON at index.commoncrawl.org/ccbot.json and documents reverse DNS verification under crawl.commoncrawl.org, so you can check a request's source two ways. GPTBot identifies itself as `GPTBot/1.4` with the URL openai.com/gptbot, and OpenAI publishes its addresses at openai.com/gptbot.json.

Use the operators' own pages for this, not a list someone compiled. Crawler lists copy each other, and an error in one spreads to many. A user-agent string proves nothing on its own: any client can claim to be CCBot or GPTBot, and only the address check tells you it was.

## How often do large sites block each one?

On 3 September 2026 we read the robots.txt files of the top 10,000 domains in Tranco list K9QPW and parsed 8,036 of them. 1,215 (15.1%) disallow CCBot at the root, the highest rate of any AI-related crawler we measured. 1,147 (14.3%) disallow GPTBot. The other side of the ledger is thinner still for CCBot: only 149 domains (1.9%) allow it by name, against 349 (4.3%) for GPTBot. And 6,672 (83.0%) never mention CCBot at all, which means their wildcard group decides for them.

The closeness of the two block rates suggests many large sites block both in the same edit, treating them as one decision about reuse. The full table and the method are in [our study of AI crawler access](/data/ai-crawler-robots-access).

## Which crawlers actually showed up on our site?

Attensira counts crawler requests at the edge on attensira.com, as it does for customers on Growth and Business. In the 30 days to 4 October 2026 the log held 143,601 AI-crawler requests, and not one was labelled CCBot. In the same window, about 11% of 7,536 requests with OpenAI user agents, roughly 830, were classed as training, which is GPTBot.

One month is a short window, and we read the absence as an absence in this window, not as evidence about how often CCBot visits sites like ours. Every label is the user agent the request claimed; none was IP-verified, and our documentation says we do not claim verified attribution. What the comparison does show is that the crawler a site worries about is not always the one that shows up.

## Is the reuse argument the same for both?

Not quite. A block on GPTBot is a statement to a company you can name, with a policy you can read. A block on CCBot is a statement about an archive whose users you cannot list. Some site owners are comfortable contributing to an open archive used by researchers and object only to commercial training by specific labs. Others treat the open archive as the main route by which their writing reaches every model, and block it first.

There is also a contribution argument for allowing CCBot. Common Crawl presents its archive as a shared resource. A site that blocks it withdraws from that commons too. That is a legitimate trade-off to make deliberately, and a poor one to make by pasting a list.

## Who should allow both?

Companies whose growth depends on being described accurately wherever a model answers without searching. A developer tool whose documentation is its product, an open-source project, a reference publisher that wants its definitions repeated: for these, being in the open archive and in a lab's training data is distribution, not loss. The trade is control over reuse for a chance, which nobody can measure for a single site, that models learn your material well. Some companies take that trade on purpose. It should be on purpose.

## Which other training crawlers belong in the same decision?

If you block CCBot and GPTBot for reasons of reuse, the same reasons usually apply to other training-side agents. In our read of 8,036 large domains, the next most blocked were Bytespider, ByteDance's crawler, at 1,150 (14.3%), ClaudeBot at 1,077 (13.4%), Meta-ExternalAgent at 994 (12.4%) and the Google-Extended token at 992 (12.3%). Large sites that block training tend to block several of these together.

Our own log shows why the list matters more than any single name. In the same 30 days, attensira.com received 131,105 requests labelled Meta-ExternalAgent across 873 pages, almost all of them (130,718) in the last 7 days, and 3,505 labelled Bytespider. A policy that named only the two crawlers on this page would have left the busiest training-side agent on our site untouched.

### Which other training crawlers do large sites block?

Two training crawlers outside this pair are blocked about as often as GPTBot in our read of 8,036 top-10,000 domains on 3 September 2026. A reuse policy that names only CCBot and GPTBot leaves them to the wildcard.

**Readings, 3 Sep 2026**

| Measure | Value |
| --- | --- |
| disallow Bytespider at the root (1,150 domains) (n=8,036 domains) | 14.3% |
| disallow Meta-ExternalAgent at the root (994 domains) (n=8,036 domains) | 12.4% |

_robots.txt of the Tranco top-10,000 (list K9QPW of 2026-09-01), read by Attensira · 3 Sep 2026_

## Can you tell from your logs whether a block is working?

Partly. After a block, a well-behaved crawler keeps fetching robots.txt and stops fetching everything else. That pattern is visible at the edge: requests from the published addresses drop to the robots file itself. If page fetches continue from those addresses, the crawler is not reading the rule you think it is, usually because of a typo in the token or a rule on the wrong host.

What logs cannot tell you is whether a block changed anything downstream. You will not see a model forget your pages, and you will not see an archive skip them until the next one is published. Treat the logs as proof that the rule is being read, and look at the answers your buyers get for everything else.

## What should you check every few months?

Read Common Crawl's CCBot page and OpenAI's crawler page again and compare them with your file: operators add agents, rename tokens and bump version strings, and a rule written for last year's list can miss this year's crawler. Look at a week of logs for requests claiming to be either crawler and check them against the published address lists. Write the reason for each training-side rule as a comment above it, so the next person who edits the file knows it was deliberate. And keep the search and user agents in their own groups, so a reuse decision never removes you from the answers your buyers read.

## FAQ

### Is CCBot an AI crawler?

It is Common Crawl's crawler, and Common Crawl is a non-profit that maintains an open repository of web crawl data. It does not train models itself. Its archive is open to anyone, and model builders are among the parties that download it, which is why CCBot appears on most lists of AI crawlers.

### If I block GPTBot, can OpenAI still get my pages from Common Crawl?

Common Crawl's archive is open to anyone who downloads it, and OpenAI's crawler page does not describe what other sources OpenAI uses. A GPTBot block governs GPTBot. If you want to keep future pages out of open datasets as well, block CCBot too.

### Does blocking either one affect ChatGPT search or Google?

No. ChatGPT search depends on OAI-SearchBot, and Google Search on Googlebot. Neither CCBot nor GPTBot is documented as a search crawler, so blocking them has no documented effect on being found in search or in AI answers that search the web.

### How do I verify CCBot?

Common Crawl publishes CCBot's address ranges at index.commoncrawl.org/ccbot.json and documents reverse DNS verification under crawl.commoncrawl.org. Its user agent is CCBot/2.0 with a commoncrawl.org URL. Check against Common Crawl's own page rather than a compiled list.

### Does blocking CCBot remove my pages from past archives?

No. A robots.txt rule affects future crawls. Pages captured in earlier Common Crawl archives remain in those archives.

### Which one do large sites block more often?

CCBot, narrowly. In our read of 8,036 top-10,000 domains on 3 September 2026, 1,215 (15.1%) disallowed CCBot at the root and 1,147 (14.3%) disallowed GPTBot. CCBot was the most often blocked AI-related crawler we measured.

## Sources

Each entry below is one claim on this page. `retrieved` is the date the exporter or a human last saw the source say this, and nothing more: it is not a publication date and not a promise of re-verification. `cite` is the anchor on the canonical page where this claim is rendered.

### ccbot-mission

Common Crawl describes its aim as democratising access to web information by maintaining an open repository of web crawl data that is universally accessible.

- source: https://commoncrawl.org/ccbot (commoncrawl.org)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-ccbot-mission
- side: ccbot

> an open repository of web crawl data that is universally accessible
### ccbot-token

CCBot is controlled with the robots.txt user-agent token CCBot.

- source: https://commoncrawl.org/ccbot (commoncrawl.org)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-ccbot-token
- side: ccbot
### ccbot-ua

Common Crawl publishes CCBot's user-agent string as CCBot/2.0 (https://commoncrawl.org/faq/)

- source: https://commoncrawl.org/ccbot (commoncrawl.org)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-ccbot-ua
- side: ccbot
### ccbot-robots

Common Crawl publishes an explicit robots.txt directive for blocking CCBot, indicating the crawler honours robots.txt.

- source: https://commoncrawl.org/ccbot (commoncrawl.org)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-ccbot-robots
- side: ccbot

> User-agent: CCBot / Disallow: /
### ccbot-ip-list

Common Crawl publishes CCBot's IPv4 and IPv6 ranges as JSON at index.commoncrawl.org/ccbot.json, and documents reverse-DNS verification against crawl.commoncrawl.org.

- source: https://commoncrawl.org/ccbot (commoncrawl.org)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-ccbot-ip-list
- side: ccbot

> These IP ranges (v4 and v6) are also provided as JSON at https://index.commoncrawl.org/ccbot.json
### ccbot-output

CCBot's output is a public archive rather than a private dataset, and Common Crawl's stated purpose is that the repository be universally accessible.

- source: https://commoncrawl.org/ccbot (commoncrawl.org)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-ccbot-output
- side: ccbot
### gptbot-purpose

OpenAI documents GPTBot as the crawler used to collect content that may be used to train its generative foundation models.

- source: https://developers.openai.com/api/docs/bots (developers.openai.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-gptbot-purpose
- side: gptbot

> Used to crawl content that may be used in training our generative AI foundation models
### gptbot-token

GPTBot is controlled with the robots.txt user-agent token GPTBot.

- source: https://developers.openai.com/api/docs/bots (developers.openai.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-gptbot-token
- side: gptbot
### gptbot-ua

OpenAI publishes GPTBot's user-agent string as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

- source: https://developers.openai.com/api/docs/bots (developers.openai.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-gptbot-ua
- side: gptbot
### gptbot-robots

OpenAI documents GPTBot as controllable through robots.txt using its own user-agent token.

- source: https://developers.openai.com/api/docs/bots (developers.openai.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-gptbot-robots
- side: gptbot
### gptbot-ip-list

OpenAI publishes GPTBot's address ranges as a machine-readable list at openai.com/gptbot.json

- source: https://developers.openai.com/api/docs/bots (developers.openai.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-gptbot-ip-list
- side: gptbot
### gptbot-output

GPTBot's output feeds OpenAI's own foundation-model training rather than a redistributable public archive.

- source: https://developers.openai.com/api/docs/bots (developers.openai.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-gptbot-output
- side: gptbot
### study-ccbot-blocked

Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 1,215 (15.1%) disallow CCBot at the site root.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-study-ccbot-blocked
- side: ccbot
### study-gptbot-blocked

Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 1,147 (14.3%) disallow GPTBot at the site root.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-study-gptbot-blocked
- side: gptbot
### study-ccbot-allowed

Of the same 8,036 domains, 149 (1.9%) allow CCBot by name and 6,672 (83.0%) never mention it.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-study-ccbot-allowed
- side: ccbot
### study-gptbot-allowed

Of the same 8,036 domains, 349 (4.3%) allow GPTBot by name and 6,540 (81.4%) never mention it.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-study-gptbot-allowed
- side: gptbot
### dog-ccbot

In the attensira.com workspace, read through the Attensira MCP on 4 October 2026, no request labelled CCBot reached attensira.com in the 30 days to 4 October 2026, out of 143,601 AI-crawler requests.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-dog-ccbot
- side: ccbot
### dog-gptbot-share

In the attensira.com workspace, read through the Attensira MCP on 4 October 2026, GPTBot was among the agents seen on all 30 of the 30 most-fetched attensira.com pages in the 30 days to 4 October 2026, by the user agent each request claimed; none was IP-verified.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/ccbot-vs-gptbot#claim-dog-gptbot-share
- side: gptbot