Skip to content
AI bots / Anthropic

ClaudeBot: Anthropic's crawler for model training

ClaudeBot is the crawler Anthropic uses to collect public web content that could help train its Claude models. When you restrict it, Anthropic says it reads that as a signal to keep your future pages out of its training data.

Anthropic runs two more agents with other jobs. So the choice is not all or nothing: you can slow this one down, block it, and still let Claude read your pages when a person asks.

The Pagereckon team Updated
Claudebot at a glance, from Pagereckon
User-agent string
ClaudeBot
Operator
Anthropic
Type
AI training crawler
robots.txt token
ClaudeBot
Obeys robots.txt
Yes, by the operator's documentation. Anthropic also supports the non-standard Crawl-delay rule
Official docs
support.claude.com

Slow ClaudeBot down instead of blocking it

Here is a detail many site owners miss. Anthropic supports Crawl-delay, a robots.txt rule that most big crawlers ignore. It sets a pause, in seconds, between requests.

robots.txt: ask for a pause between requests
User-agent: ClaudeBot
Crawl-delay: 1

Anthropic says it aims to respect the delay where appropriate. As a result, a small shared host that struggles with crawl bursts can try this line first, before cutting the crawler off.

How to block ClaudeBot in WordPress

To opt out of training, add a Disallow for the token. Anthropic asks you to repeat the rule on every subdomain you want to cover, since each one serves its own robots.txt.

robots.txt: opt out of Anthropic training
User-agent: ClaudeBot
Disallow: /

Anthropic warns against IP blocks. It says that blocking its addresses may not give a lasting opt-out, because the crawler then cannot read your robots.txt. That advice matters, since a server rule based on the user agent has a similar effect.

If you still want a hard stop for a copycat that uses the name, these rules answer 403:

.htaccess (Apache, LiteSpeed)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
RewriteRule .* - [F,L]
</IfModule>
Nginx server block
if ($http_user_agent ~* "ClaudeBot") {
    return 403;
}

Pagereckon Pro offers one-click blocking for bots that crawl hard and give nothing back, such as Bytespider. This crawler follows robots.txt, so it does not get that button. The robots.txt line is the method Anthropic itself asks for.

What ClaudeBot does not control

Blocking the training crawler does not stop Claude-User, which fetches a page when a person asks Claude about it. It also leaves Claude-SearchBot alone, which indexes pages for Claude's search answers.

Anthropic's three agents
AgentJobEffect of a Disallow
ClaudeBotCollects content for trainingFuture pages left out of training
Claude-SearchBotIndexes for search qualityPages not indexed for Claude search
Claude-UserFetches for a user's questionClaude cannot read the page for users

So the cost of blocking ClaudeBot alone is small in traffic terms. A training crawl sends no readers by itself.

Verify a request that claims to be ClaudeBot

Anthropic does not print a full user agent string on its help page, only the token. To check a visit, use the IP address instead. Anthropic publishes its crawler addresses at claude.com/crawling/bots.json, and a source IP on that list means the request came from Anthropic.

When an IP does not match, the visitor only borrowed the name. That is where a firewall or server rule makes sense.

ClaudeBot in the Pagereckon AI report

Pagereckon matches the token in your access log and files these hits under training, apart from Claude's search and live read traffic. You see which posts get read and how the volume changes week to week.

Because Pagereckon reads the server log, it also catches requests that never run JavaScript, which is every crawler. Open the AI crawler checker to see the split for your own site, or compare agents on the AI bots list.

ClaudeBot questions

Does ClaudeBot respect robots.txt?

Yes. Anthropic says its bots honor robots.txt directives, and it also supports Crawl-delay.

Will blocking ClaudeBot stop Claude from reading my site?

No. Claude-User and Claude-SearchBot have their own tokens. Block those too if you want Claude to stay away completely.

Should I block Anthropic's IP addresses?

Anthropic advises against it, because an IP block stops the crawler from reading your robots.txt. Use a Disallow instead.

Does the free Pagereckon plugin show ClaudeBot?

Yes. AI crawlers per page are part of the free plugin. Download the free plugin to see them.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.