CCBot: the crawler behind Common Crawl's open archive
CCBot is the crawler that Common Crawl, a non-profit foundation, uses to build an open repository of web crawl data. Anyone can download and analyze that archive, which is the whole point of the project.
That openness is also why site owners ask about it. When a crawl is free for everyone, you cannot know every company that will later use the copy of your pages.
CCBot/2.0 (https://commoncrawl.org/faq/)
- Operator
- Common Crawl
- Type
- Several purposes
- robots.txt token
CCBot- Obeys robots.txt
- Yes, by the operator's documentation. Common Crawl also obeys Crawl-delay and honors link-level nofollow
- Official docs
- commoncrawl.org
Why CCBot is different from company crawlers
GPTBot or ClaudeBot collect pages for one company's own models. In contrast, Common Crawl publishes what it collects as open data for research and analysis. So a single decision about this crawler affects every downstream user of the archive at once.
Common Crawl also says its dataset is a sample of the web. It does not usually archive whole sites, only a randomly chosen subset of pages.
| Topic | What Common Crawl says |
|---|---|
| robots.txt | Checked first; pages fetched only when allowed |
| Crawl-delay | Obeyed |
| nofollow links | Honored |
| Sitemaps | Read when announced in robots.txt |
| JavaScript and cookies | Not executed, not used |
Slow CCBot down or block it
If the crawl rate is the only problem, ask for a pause first. Common Crawl says it obeys Crawl-delay, and a higher number tells the crawler to slow down.
User-agent: CCBot
Crawl-delay: 10To keep it out completely, use the rule Common Crawl gives on its own page:
User-agent: CCBot
Disallow: /Since the crawler checks robots.txt before it fetches pages, you need no server rule for the real one. Likewise, Pagereckon Pro's one-click blocking targets bots that crawl hard and ignore the rules, such as Bytespider, so it does not cover this crawler.
Fake CCBot traffic, and how to check
Common Crawl warns that some crawlers falsely claim to be CCBot. For that reason it now runs on dedicated IP ranges with reverse DNS.
- Look up the IP from your log with a reverse DNS query.
- The host name should end in
crawl.commoncrawl.org. - Then resolve that host name and confirm it returns the same IP.
Reverse DNS does not work over IPv6 yet, per Common Crawl. For those requests, match the address against the published list at index.commoncrawl.org/ccbot.json instead. A copycat that fails both checks ignores your robots.txt anyway, so a server rule is the right answer for it.
What blocking CCBot costs
Common Crawl is not a search engine, so a block does not cost you rankings or clicks. The trade is about reach of a different kind: your pages leave a public archive that researchers and builders draw on.
Whether that is a loss depends on your goals. A site that wants its writing in open datasets keeps the crawler. A site that wants tight control over reuse blocks it.
CCBot in Pagereckon
Pagereckon lists CCBot among the AI crawlers, outside the per-assistant split, since it serves no single assistant. You see which pages it fetched and when, straight from your server log.
Next, compare it with other agents in the AI crawler checker, or read about each one on the AI bots list.
CCBot questions
What is CCBot?
It is the crawler of Common Crawl, a non-profit that publishes an open repository of web crawl data.
Does CCBot respect robots.txt?
Yes. Common Crawl says it checks robots.txt before fetching pages, obeys Crawl-delay and honors nofollow links.
How can I tell a fake CCBot?
Do a reverse DNS lookup. A real one resolves under crawl.commoncrawl.org, or its IP appears in ccbot.json.
Is tracking it free?
Yes. Download the free plugin and CCBot shows up with the other AI crawlers.