ClaudeBot: Anthropic's crawler for model training
ClaudeBot is the crawler Anthropic uses to collect public web content that could help train its Claude models. When you restrict it, Anthropic says it reads that as a signal to keep your future pages out of its training data.
Anthropic runs two more agents with other jobs. So the choice is not all or nothing: you can slow this one down, block it, and still let Claude read your pages when a person asks.
ClaudeBot
- Operator
- Anthropic
- Type
- AI training crawler
- robots.txt token
ClaudeBot- Obeys robots.txt
- Yes, by the operator's documentation. Anthropic also supports the non-standard Crawl-delay rule
- How to verify
- Match the IP against claude.com/crawling/bots.json
- Official docs
- support.claude.com
Slow ClaudeBot down instead of blocking it
Here is a detail many site owners miss. Anthropic supports Crawl-delay, a robots.txt rule that most big crawlers ignore. It sets a pause, in seconds, between requests.
User-agent: ClaudeBot
Crawl-delay: 1Anthropic says it aims to respect the delay where appropriate. As a result, a small shared host that struggles with crawl bursts can try this line first, before cutting the crawler off.
How to block ClaudeBot in WordPress
To opt out of training, add a Disallow for the token. Anthropic asks you to repeat the rule on every subdomain you want to cover, since each one serves its own robots.txt.
User-agent: ClaudeBot
Disallow: /Anthropic warns against IP blocks. It says that blocking its addresses may not give a lasting opt-out, because the crawler then cannot read your robots.txt. That advice matters, since a server rule based on the user agent has a similar effect.
If you still want a hard stop for a copycat that uses the name, these rules answer 403:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
RewriteRule .* - [F,L]
</IfModule>if ($http_user_agent ~* "ClaudeBot") {
return 403;
}Pagereckon Pro offers one-click blocking for bots that crawl hard and give nothing back, such as Bytespider. This crawler follows robots.txt, so it does not get that button. The robots.txt line is the method Anthropic itself asks for.
What ClaudeBot does not control
Blocking the training crawler does not stop Claude-User, which fetches a page when a person asks Claude about it. It also leaves Claude-SearchBot alone, which indexes pages for Claude's search answers.
| Agent | Job | Effect of a Disallow |
|---|---|---|
| ClaudeBot | Collects content for training | Future pages left out of training |
| Claude-SearchBot | Indexes for search quality | Pages not indexed for Claude search |
| Claude-User | Fetches for a user's question | Claude cannot read the page for users |
So the cost of blocking ClaudeBot alone is small in traffic terms. A training crawl sends no readers by itself.
Verify a request that claims to be ClaudeBot
Anthropic does not print a full user agent string on its help page, only the token. To check a visit, use the IP address instead. Anthropic publishes its crawler addresses at claude.com/crawling/bots.json, and a source IP on that list means the request came from Anthropic.
When an IP does not match, the visitor only borrowed the name. That is where a firewall or server rule makes sense.
ClaudeBot in the Pagereckon AI report
Pagereckon matches the token in your access log and files these hits under training, apart from Claude's search and live read traffic. You see which posts get read and how the volume changes week to week.
Because Pagereckon reads the server log, it also catches requests that never run JavaScript, which is every crawler. Open the AI crawler checker to see the split for your own site, or compare agents on the AI bots list.
ClaudeBot questions
Does ClaudeBot respect robots.txt?
Yes. Anthropic says its bots honor robots.txt directives, and it also supports Crawl-delay.
Will blocking ClaudeBot stop Claude from reading my site?
No. Claude-User and Claude-SearchBot have their own tokens. Block those too if you want Claude to stay away completely.
Should I block Anthropic's IP addresses?
Anthropic advises against it, because an IP block stops the crawler from reading your robots.txt. Use a Disallow instead.
Does the free Pagereckon plugin show ClaudeBot?
Yes. AI crawlers per page are part of the free plugin. Download the free plugin to see them.