Skip to content
AI bots / Common Crawl

CCBot: the crawler behind Common Crawl's open archive

CCBot is the crawler that Common Crawl, a non-profit foundation, uses to build an open repository of web crawl data. Anyone can download and analyze that archive, which is the whole point of the project.

That openness is also why site owners ask about it. When a crawl is free for everyone, you cannot know every company that will later use the copy of your pages.

The Pagereckon team Updated
Ccbot at a glance, from Pagereckon
User-agent string
CCBot/2.0 (https://commoncrawl.org/faq/)
Operator
Common Crawl
Type
Several purposes
robots.txt token
CCBot
Obeys robots.txt
Yes, by the operator's documentation. Common Crawl also obeys Crawl-delay and honors link-level nofollow
Official docs
commoncrawl.org

Why CCBot is different from company crawlers

GPTBot or ClaudeBot collect pages for one company's own models. In contrast, Common Crawl publishes what it collects as open data for research and analysis. So a single decision about this crawler affects every downstream user of the archive at once.

Common Crawl also says its dataset is a sample of the web. It does not usually archive whole sites, only a randomly chosen subset of pages.

How CCBot behaves, per Common Crawl's docs
TopicWhat Common Crawl says
robots.txtChecked first; pages fetched only when allowed
Crawl-delayObeyed
nofollow linksHonored
SitemapsRead when announced in robots.txt
JavaScript and cookiesNot executed, not used

Slow CCBot down or block it

If the crawl rate is the only problem, ask for a pause first. Common Crawl says it obeys Crawl-delay, and a higher number tells the crawler to slow down.

robots.txt: ask for a 10 second pause
User-agent: CCBot
Crawl-delay: 10

To keep it out completely, use the rule Common Crawl gives on its own page:

robots.txt: block Common Crawl
User-agent: CCBot
Disallow: /

Since the crawler checks robots.txt before it fetches pages, you need no server rule for the real one. Likewise, Pagereckon Pro's one-click blocking targets bots that crawl hard and ignore the rules, such as Bytespider, so it does not cover this crawler.

Fake CCBot traffic, and how to check

Common Crawl warns that some crawlers falsely claim to be CCBot. For that reason it now runs on dedicated IP ranges with reverse DNS.

  1. Look up the IP from your log with a reverse DNS query.
  2. The host name should end in crawl.commoncrawl.org.
  3. Then resolve that host name and confirm it returns the same IP.

Reverse DNS does not work over IPv6 yet, per Common Crawl. For those requests, match the address against the published list at index.commoncrawl.org/ccbot.json instead. A copycat that fails both checks ignores your robots.txt anyway, so a server rule is the right answer for it.

What blocking CCBot costs

Common Crawl is not a search engine, so a block does not cost you rankings or clicks. The trade is about reach of a different kind: your pages leave a public archive that researchers and builders draw on.

Whether that is a loss depends on your goals. A site that wants its writing in open datasets keeps the crawler. A site that wants tight control over reuse blocks it.

CCBot in Pagereckon

Pagereckon lists CCBot among the AI crawlers, outside the per-assistant split, since it serves no single assistant. You see which pages it fetched and when, straight from your server log.

Next, compare it with other agents in the AI crawler checker, or read about each one on the AI bots list.

CCBot questions

What is CCBot?

It is the crawler of Common Crawl, a non-profit that publishes an open repository of web crawl data.

Does CCBot respect robots.txt?

Yes. Common Crawl says it checks robots.txt before fetching pages, obeys Crawl-delay and honors nofollow links.

How can I tell a fake CCBot?

Do a reverse DNS lookup. A real one resolves under crawl.commoncrawl.org, or its IP appears in ccbot.json.

Is tracking it free?

Yes. Download the free plugin and CCBot shows up with the other AI crawlers.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.