CCBot vs GPTBot: the open web archive against one vendor's training crawler

The short answer

CCBot and GPTBot both collect web pages that end up in language models, and they get there by completely different routes. GPTBot collects content for OpenAI's own foundation-model training. CCBot fills a public archive that Common Crawl describes as universally accessible, which anyone — including every model builder who has not written their own crawler — can download.

That difference is why a publisher who disallows GPTBot and stops there has done much less than they think.

One crawler, many consumers

Common Crawl's stated purpose is democratising access to web information by maintaining an open repository of web crawl data. The archive is the product. It is used by academic researchers, by search startups, by linguists, by people building small models on a budget, and by large model builders who prefer not to run their own crawl.

That plurality is the whole argument in both directions. If you believe the open web should have an open archive — that the ability to study the web at scale should not belong exclusively to companies who can afford to crawl it — CCBot is the crawler you want to allow, and it is the one whose absence hurts small players far more than large ones. A large model builder can and does run its own crawler. A researcher cannot.

If instead your objection is that your work should not be redistributed into corpora you cannot enumerate, CCBot is the higher-leverage block precisely for the same reason: one directive, many downstream consumers. Blocking GPTBot removes you from one company's intake. Blocking CCBot removes you from a supply chain.

Both positions are defensible. What is not defensible is holding the second objection and writing only the first rule.

What each block actually reaches

There is a timing problem that no robots directive solves, and it is sharper for CCBot than for anything else on this site. Common Crawl has been archiving for far longer than the current wave of model-training crawlers has existed. A robots.txt rule you add today governs future crawls. It does not retract snapshots already published, and those snapshots have already been downloaded by parties you will never have a list of.

GPTBot has the same limitation in principle — a rule today does not unlearn a model trained yesterday — but the shape is different. OpenAI is one counterparty with published documentation and a stated policy. Common Crawl's archive is an artefact in the world. This is not a criticism of Common Crawl, which has been transparent about what it does since long before anybody was upset about it. It is simply the honest answer to "will blocking this remove my content", and the honest answer is: from future crawls, yes; from what already exists, no.

Copyable robots.txt directives

Refuse the open archive, allow the vendor crawler — an unusual position, but coherent for a publisher who has a commercial relationship with one model provider:

User-agent: CCBot
Disallow: /

Refuse both, which is what a publisher with a general objection to training corpora actually wants:

User-agent: CCBot
Disallow: /

User-agent: GPTBot
Disallow: /

Refuse the archive for a valuable section only, leaving the marketing site open:

User-agent: CCBot
Disallow: /research/
Disallow: /reports/

Note what none of these do: they do not touch retrieval. OpenAI's OAI-SearchBot and its equivalents at other vendors are separate tokens, and none of the blocks above affect whether an assistant can fetch and cite your page when a person asks about you. Keeping that distinction straight is the single most valuable habit in this whole family.

Telling them apart in logs

CCBot's published user agent is short and stable: CCBot/2.0 (https://commoncrawl.org/faq/). GPTBot's is long and versioned, ending in compatible; GPTBot/1.4; +https://openai.com/gptbot. Both are easy to match on. Neither is proof of anything on its own, because a user-agent header is self-reported.

Verification is the place a user-agent string stops being enough, and both operators know it. OpenAI publishes GPTBot's address ranges as a machine-readable list at openai.com/gptbot.json. Common Crawl publishes CCBot's IPv4 and IPv6 prefixes at index.commoncrawl.org/ccbot.json and goes further, running CCBot on dedicated ranges with reverse DNS so a logged address resolves under crawl.commoncrawl.org — it recommends checking, in as many words, because spoofed CCBots exist.

The asymmetry is narrower than it first looks, and it runs the other way on IPv4: only one of the two documents a reverse-DNS path. It reappears on IPv6, where Common Crawl notes reverse DNS is not yet supported, leaving the published prefix list as the only check. Either way, do not write a firewall rule on the user-agent string alone. Both operators give you something better, and both expect you to use it.

The version numbers cut the other way. CCBot has published a 2.0 version consistently, while GPTBot has already moved to 1.4. A log filter or rule matching GPTBot/1.0 will silently stop working after a version bump; the same style of filter written for CCBot is far more durable. Match on the token, not on the version, in both cases.

What Common Crawl is actually used for

It is easy to talk about CCBot as though its only downstream consumer were a large model builder, and that framing quietly loads the argument. The archive has a much wider constituency, and a publisher deciding whether to block it should know who else is in the room.

Common Crawl's stated purpose is democratising access to web information — maintaining a repository that is universally accessible rather than one held privately. In practice that repository is a research instrument. It is how somebody studies the evolution of web technology at scale without running their own global crawl, how a linguist assembles a corpus in an under-resourced language, how a small search startup gets a seed index without a budget for infrastructure, and how an academic reproduces somebody else's result on the same data.

None of those parties could crawl the web themselves. The large model builders can, and several of them do. So the practical effect of a widespread CCBot block is not to stop training. It is to widen the gap between the organisations that can afford a private crawl and everyone else, while the private crawls continue.

That is not an argument that you must allow CCBot. It is an argument that the trade-off is not the one it looks like from the outside. If your objection is to concentrated corporate power over training data, blocking the open archive is close to the opposite of an effective response. If your objection is that your specific work should not be redistributed at all, then it is exactly the right response, and the cost to researchers is a cost you have decided to accept knowingly rather than by accident.

The reversibility problem

There is one more asymmetry between these two crawlers, and it is the one that should make a publisher think hardest before writing a CCBot rule: these decisions are not equally easy to undo.

Unblocking GPTBot restores OpenAI's ability to crawl you, and future crawls resume. The relationship is with one company, whose policy is published, and the state of it is whatever your current file says.

Unblocking CCBot restores future archiving. It does not fill in the gap. A site absent from the archive for two years has a two-year hole in the public record of the web, and that hole is permanent in every snapshot already published and downloaded. For a company with a long-lived site, that hole is also a hole in your own history — the archive is one of the few places a page you deleted in 2023 might still be findable.

Which cuts both ways, of course. If your reason for blocking is that you do not want a durable public copy of your content to exist, the durability is precisely the thing you are objecting to. Just make the decision with the permanence in view, rather than treating a CCBot line as a directive you can revisit next quarter with no residue.

Making the decision

Three questions, and they resolve most cases.

Do you sell the content itself, or does the content sell something else? A research firm, a publisher, a paid archive has a direct commercial interest in not being redistributed. A software company's documentation has the opposite interest: being in the corpus is how a model comes to know your product exists.

Is your objection to a company or to a practice? If it is to a company, the vendor token is the right instrument and the archive block is collateral damage against researchers you have no quarrel with. If it is to the practice, the archive block is the one that matters and the vendor token alone is theatre.

Can you live with the historical archive? If not, robots.txt is not the tool you need, and no amount of it will become the tool you need.

Once you have decided, verify. Attensira's crawler logs record which agent fetched which URL and when, so you can see whether CCBot and GPTBot actually stopped, rather than trusting that they did. Before you edit, the bot access score will read back what your current file permits, and the robots.txt generator will assemble a correct one.

Related comparisons

For the OpenAI token that governs retrieval rather than training — the one you almost certainly want to leave alone — read GPTBot vs OAI-SearchBot. Meta's training crawler makes an instructive third comparison in Meta-ExternalAgent vs GPTBot. Google's approach of governing use rather than access is covered in Google-Extended vs Googlebot, and Anthropic's three-token split in ClaudeBot vs Claude-User.