Bots
What is CCBot, and should you block it?
CCBot is the crawler of Common Crawl, a non-profit that publishes an open archive of the web anyone can analyse. Common Crawl does not train models itself, but its archive is a common starting point for AI training data. Block it if you want out of future archives; blocking costs no AI citations and does not remove copies already published.
7 days free · card required
0requests filed as CCBot · 30 days
of 143,601 AI crawler requests from 7 operators
We name CCBot and disallow the whole site. Nothing else in the file changes for it.
User-agent: CCBotDisallow: /
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated Who runs CCBot, and what is it for?
Common Crawl Foundation runs it, and we file it under training. Builds a public web archive that AI training datasets are drawn from. It is not a search index, so blocking it removes no page from any assistant's search results or citations.
From Common Crawl Foundation’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
How do I allow or block CCBot in robots.txt?
Paste this into the robots.txt at the root of your domain.
# Keep this site out of future# Common Crawl archives.User-agent: CCBotDisallow: /Is CCBot an AI training crawler?
Not in its operator's words, but in effect. CCBot is run by the Common Crawl Foundation, a non-profit whose stated goal is democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone1. Common Crawl does not train models. It collects and publishes an archive.
Because the archive is public, anyone assembling a training corpus can start from it, and Common Crawl does not decide who does. That is why a foundation with an open-data mission ends up in a discussion about AI training access. This page classifies CCBot's job as training. The classification is ours, not Common Crawl's wording: it describes where the data ends up and so what allowing this crawler costs you. A training fetch never ends in a link, so blocking CCBot costs no citations or referrals; text already in the archive stays there.
Why does a training opt-out need a CCBot group?
Because a file built from a list of model companies stops at the model companies. A robots.txt written that way names GPTBot, ClaudeBot, perhaps Meta-ExternalAgent, and ends. That covers first-party training crawlers and leaves the open archive many corpora are derived from untouched.
The study figures below show the sites that did think about it. In our robots.txt run across the top 10,000 Tranco domains on 3 September 2026, CCBot had the highest block rate of any crawler measured: 1,215 of 8,036 readable files, about 15.1%, ahead of Bytespider (1,150) and GPTBot (1,147). Put the other way, the other 6,821 readable files leave CCBot free to crawl, and a site that blocks GPTBot but not CCBot has kept one company out and left the archive open. If you have written a training-exclusion file, check it for this token. attensira.com's own robots.txt disallows CCBot by name, the one training crawler it blocks. The archive is handed on to anyone who downloads it, with no attribution back to the site. GPTBot and ClaudeBot stay allowed by name in the same file.
Common Crawl publishes the block itself, on the same page as its user agent: a group naming CCBot with Disallow: / prevents Common Crawl from crawling a site3. It is in the robots.txt section above.
Agents filed beside CCBot
How do I verify that a visit really came from CCBot?
A user-agent header is free text, so a log line naming CCBot is a claim until you check it.
Look up the visiting IP, confirm the hostname ends in one of the suffixes in the record, then look that hostname up and confirm it returns the same IP, or match the IP against the published ranges.
Common Crawl says crawlers falsely identify as CCBot and recommends verifying. Over IPv6 reverse DNS is not yet supported, so an empty v6 lookup is that gap, not proof of a fake; use the JSON ranges there. A heavy stream that does not look like a periodic archive crawl and fails both checks is a scraper, and your CCBot rule never applied to it.
What can a CCBot block not do?
Two limits matter. First, the rule is forward-looking. Archives already published cannot be unpublished by a rule you add today, and datasets derived from them are copies that already exist. Second, robots.txt is not Common Crawl's only mechanism. It links an opt-out ledger and registry from its own site5, a separate process worth reading if exclusion actually matters to you.
How is CCBot different from GPTBot?
Operator
- CCBot
- Common Crawl Foundation (non-profit)
- GPTBot
- OpenAI
What the crawl produces
- CCBot
- A public archive anyone can use
- GPTBot
- Content that may train OpenAI's models
Who uses the data
- CCBot
- Anyone who downloads the archive
- GPTBot
- OpenAI
Effect of blocking on AI citations
- CCBot
- None
- GPTBot
- None
The decision is similar; the reach is not. A GPTBot rule addresses one company. A CCBot rule addresses everyone who builds on the archive in future. The full comparison is in CCBot vs GPTBot.
- Reverse DNS ends in
- *.crawl.commoncrawl.org
- IP range file
- ccbot.json
ResearcherTranscribed from Common Crawl Foundation documentation · How often do sites block CCBot?
Measured by us, not reported by Common Crawl Foundation: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow CCBot at the root
- 15.1%
- 1,215 of 8,036 domains
- allow it by name
- 1.8%
- 149 domains
- never mention it
- 83.0%
- 6,672 domains
How should you match CCBot in logs?
Common Crawl documents the header as CCBot/2.02. The 2.0 has a version and can change, so match the bare token CCBot, case-insensitively, as a whole word.
What can Attensira show you about CCBot?
AI traffic, on Growth and Business, counts crawler fetches per operator and declared purpose at the edge, so archive crawling sits apart from the agents that fetch for live answers. It reports user agents as sent, and the verification checks stay yours. The robots.txt generator lists CCBot among the agents you can block, and crawler logs shows what keeps arriving afterwards.
CCBot/2.0 (https:/Should I allow CCBot in my situation?
Concrete cases, and what each one implies for this agent.
You want out of AI training corpora and you have already blocked the model companies' training crawlers.
Add a CCBot group as well.
Blocking GPTBot and its equivalents addresses first-party training crawls. It does nothing about a corpus assembled from an open archive that a different organisation collected. CCBot is that archive's collector, and it is the single most commonly missed token in a training-exclusion robots.txt file.
Your logs are full of CCBot hits at a volume that does not look like a periodic archive crawl.
Verify before you act — reverse DNS on the address, or match it against ccbot.json.
Common Crawl says outright that it is aware of crawlers falsely identifying themselves as CCBot and recommends verifying. A spoofed CCBot is not bound by the CCBot rule you wrote, so rate-limiting the real one while the impostor keeps working is the worst of both outcomes.
You are on IPv6 and the reverse-DNS check comes back with nothing.
Fall back to the published ranges in ccbot.json.
Common Crawl states that reverse DNS is not yet supported over IPv6. An empty PTR record there is not evidence of a fake; it is the documented gap, and the JSON file covers both v4 and v6.
What else do people ask?
Does Common Crawl train AI models?
I blocked GPTBot and ClaudeBot. Am I out of AI training?
How many sites block CCBot?
Where do these facts come from?
5 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Common Crawl describes itself as a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.
commoncrawl.org · retrieved
“democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone”
Common Crawl documents that CCBot identifies itself in its user-agent string as "CCBot/2.0 (https://commoncrawl.org/faq/)".
commoncrawl.org · retrieved · changes often, check the source
Common Crawl documents the robots.txt block for CCBot as a group naming the CCBot user agent with Disallow "/", to prevent Common Crawl from crawling a website.
commoncrawl.org · retrieved
“To prevent Common Crawl from crawling your website, include the following in your robots.txt”
Common Crawl states that CCBot is run on dedicated IP address ranges with reverse DNS (except over IPv6, where reverse DNS is not yet supported), that these IPv4 and IPv6 ranges are also provided as JSON at index.commoncrawl.org/ccbot.json, and that it is aware of crawlers falsely identifying themselves as CCBot.
commoncrawl.org · retrieved
“Please note that we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity.”
Common Crawl links an opt-out ledger and registry from its own site alongside the CCBot documentation.
commoncrawl.org · retrieved
Want whatever it takes to win the answer?
7 days free · card required