Bots
Web archive crawlers
The archiver category holds crawlers that build public web archives others reuse. CCBot is Common Crawl's crawler, and its open archive is the source many AI training datasets are derived from. Common Crawl says CCBot respects robots.txt and publishes reverse DNS and IP ranges to verify it. A Disallow keeps your pages out of future crawls.
7 days free · card required
- Operator
- Common Crawl Foundation
- What the fetch is for
- Training
- robots.txt
- Respected
- Verify a visit by
- IP ranges + reverse DNS
- attensira.com/robots.txt
- Blocked site-wide
- Requests to attensira.com, 30 days
- 0 filed as CCBot
ResearcherCommon Crawl Foundation documentation, our AI traffic board (n=143,601) and robots.txt · Which archivers are documented?
Builds a public web archive that AI training datasets are drawn from. It is not a search index, so blocking it removes no page from any assistant's search results or citations.
Keep this site out of future Common Crawl archives.
User-agent: CCBotDisallow: /Did CCBot fetch attensira.com in the last 30 days?
No request in the window was filed as CCBot, out of every AI request the board counted. User agents as sent, not checked against address lists.
We name CCBot and disallow the whole site. Nothing else in the file changes for it.
- requests filed as CCBot, 30 days
- 0
- of n=143,601 AI requests
- requests from any Common Crawl Foundation agent
- 0
- 7 operators on the board, Common Crawl Foundation not among them
Why is a web archive a training question?
CCBot does not train anything itself. It builds Common Crawl's open archive, and many AI training sets are derived from that archive. Blocking CCBot is therefore how a site keeps its future pages out of datasets built from Common Crawl, independently of each AI lab's own training crawler.
robots.txt applies when a crawler fetches. It does not reach back into archives already published, or into datasets already built from them.
- Reverse DNS ends in
- *.crawl.commoncrawl.org
- IP range file
- ccbot.json
Common Crawl says crawlers falsely identify as CCBot and recommends verifying. Over IPv6 reverse DNS is not yet supported, so an empty v6 lookup is that gap, not proof of a fake; use the JSON ranges there. A heavy stream that does not look like a periodic archive crawl and fails both checks is a scraper, and your CCBot rule never applied to it. The CCBot entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Common Crawl Foundation documentation · Does blocking CCBot cost AI search visibility?
CCBot builds an archive, not a search index, so blocking it does not take you out of any assistant's search results. Those come from the search-indexing crawlers and user-triggered fetchers, which have their own tokens and their own categories here.
attensira.com blocks CCBot site-wide while allowing the AI search crawlers by name. A bulk archive gives us no attribution, and refusing it costs nothing we can measure.
How often do the most visited sites block web archive crawlers?
- of sites disallow CCBot at the root
- 15.1%
- 1,215 of 8,036 robots.txt files
- name it with an explicit allow
- 149
- of 8,036
Web engineerRobots.txt access study, our own fetch of the top 10,000 domains (Tranco) · as of CCBot is disallowed at the site root by 1,215 of the 8,036 sites whose robots.txt we read (15.1%). Another 6,672 never mention it at all.
A site that never names an agent is counted as unspecified, not as allowing it.
What else do people ask?
What is an archiver bot?
Does blocking CCBot remove my pages from existing datasets?
Does blocking CCBot affect ChatGPT or Perplexity answers?
How do I verify CCBot?
Where do these facts come from?
One claim on this page, with the page it was read from; last checked .
Common Crawl describes itself as a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.
commoncrawl.org · retrieved
“democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone”
Want whatever it takes to win the answer?
7 days free · card required