CCBot vs GPTBot: the open web archive against one lab's training crawler

Why does CCBot appear on lists of AI crawlers if it does not train models?

Because its output is one of the most widely used sources of web text. Common Crawl publishes its data in formats designed for reuse and encourages research on it. Model builders are among the users. So CCBot sits on the training side of a robots.txt policy even though Common Crawl itself trains nothing: it is the upstream of many downstream uses you cannot see or list.

That makes CCBot the broadest single training-side decision you can write in robots.txt. A GPTBot rule covers one named crawler from one named lab. A CCBot rule covers every party that would otherwise have got your future pages from the open archive.

What does each block reach, and what does it miss?

A GPTBot block reaches GPTBot. OpenAI's crawler page does not describe the company's other data sources, so a GPTBot rule does not, by itself, tell you whether OpenAI could obtain the same pages another way. A CCBot block reaches future Common Crawl archives. It does not reach past archives, which remain available, and it does not reach anyone who crawls the web with their own crawler.

Neither block touches search. ChatGPT search depends on OAI-SearchBot. Google Search depends on Googlebot. Perplexity depends on PerplexityBot. If you write CCBot and GPTBot rules and nothing else, every answer engine that searches the web can still read, cite and link your pages.

Answer engines describe GPTBot the same narrow way its operator does. Ask one what GPTBot is and you get OpenAI's training crawler, robots.txt control and the opt-out. You do not get the other route by which the same page can reach a training set, which is the one CCBot opens.

How do you verify each crawler?

CCBot identifies itself as CCBot/2.0 with the URL commoncrawl.org/faq. Common Crawl publishes its IP ranges as JSON at index.commoncrawl.org/ccbot.json and documents reverse DNS verification under crawl.commoncrawl.org, so you can check a request's source two ways. GPTBot identifies itself as GPTBot/1.4 with the URL openai.com/gptbot, and OpenAI publishes its addresses at openai.com/gptbot.json.

Use the operators' own pages for this, not a list someone compiled. Crawler lists copy each other, and an error in one spreads to many. A user-agent string proves nothing on its own: any client can claim to be CCBot or GPTBot, and only the address check tells you it was.

How often do large sites block each one?

On 3 September 2026 we read the robots.txt files of the top 10,000 domains in Tranco list K9QPW and parsed 8,036 of them. 1,215 (15.1%) disallow CCBot at the root, the highest rate of any AI-related crawler we measured. 1,147 (14.3%) disallow GPTBot. The other side of the ledger is thinner still for CCBot: only 149 domains (1.9%) allow it by name, against 349 (4.3%) for GPTBot. And 6,672 (83.0%) never mention CCBot at all, which means their wildcard group decides for them.

The closeness of the two block rates suggests many large sites block both in the same edit, treating them as one decision about reuse. The full table and the method are in our study of AI crawler access.

Is the reuse argument the same for both?

Not quite. A block on GPTBot is a statement to a company you can name, with a policy you can read. A block on CCBot is a statement about an archive whose users you cannot list. Some site owners are comfortable contributing to an open archive used by researchers and object only to commercial training by specific labs. Others treat the open archive as the main route by which their writing reaches every model, and block it first.

There is also a contribution argument for allowing CCBot. Common Crawl presents its archive as a shared resource. A site that blocks it withdraws from that commons too. That is a legitimate trade-off to make deliberately, and a poor one to make by pasting a list.

Who should allow both?

Companies whose growth depends on being described accurately wherever a model answers without searching. A developer tool whose documentation is its product, an open-source project, a reference publisher that wants its definitions repeated: for these, being in the open archive and in a lab's training data is distribution, not loss. The trade is control over reuse for a chance, which nobody can measure for a single site, that models learn your material well. Some companies take that trade on purpose. It should be on purpose.

Can you tell from your logs whether a block is working?

Partly. After a block, a well-behaved crawler keeps fetching robots.txt and stops fetching everything else. That pattern is visible at the edge: requests from the published addresses drop to the robots file itself. If page fetches continue from those addresses, the crawler is not reading the rule you think it is, usually because of a typo in the token or a rule on the wrong host.

What logs cannot tell you is whether a block changed anything downstream. You will not see a model forget your pages, and you will not see an archive skip them until the next one is published. Treat the logs as proof that the rule is being read, and look at the answers your buyers get for everything else.

What should you check every few months?

Read Common Crawl's CCBot page and OpenAI's crawler page again and compare them with your file: operators add agents, rename tokens and bump version strings, and a rule written for last year's list can miss this year's crawler. Look at a week of logs for requests claiming to be either crawler and check them against the published address lists. Write the reason for each training-side rule as a comment above it, so the next person who edits the file knows it was deliberate. And keep the search and user agents in their own groups, so a reuse decision never removes you from the answers your buyers read.