Claude SearchBot: how Claude finds pages to cite
Claude SearchBot is the Anthropic crawler that reads the web to improve the quality and relevance of Claude's search results. If ClaudeBot is about training, this one is about being found when someone searches through Claude.
Anthropic introduced it as a separate token, so you can treat search and training differently. Below is what that choice means for a WordPress site.
Claude-SearchBot
- Operator
- Anthropic
- Type
- AI search indexer
- robots.txt token
Claude-SearchBot- Obeys robots.txt
- Yes, by the operator's documentation
- How to verify
- Match the IP against claude.com/crawling/bots.json
- Official docs
- support.claude.com
Search, not training
Anthropic gives each of its three agents one job. First, this crawler analyzes pages to make search answers more relevant and accurate. Training data comes from ClaudeBot, and live reads for one question come from Claude-User.
So the usual setup for publishers who want citations but not training looks like this:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Since Anthropic says its bots honor robots.txt, those few lines are all you need. There is no plugin and no server rule involved.
What blocking Claude SearchBot costs
Anthropic spells out the trade. If you disable the crawler, its system cannot index your content for search, and your site may lose visibility and accuracy in Claude's search results.
In practice, this means Claude may cite another page on your topic instead of yours. It may also describe your product from older or second-hand sources. For a site that wants to be the reference on a topic, that is a real cost.
User-agent: Claude-SearchBot
Disallow: /We leave out a server rule here on purpose. Anthropic warns that IP blocks stop its bots from reading robots.txt, which can undo a clean opt-out. Likewise, Pagereckon Pro's one-click blocking only covers bots that crawl hard and give nothing back, such as Bytespider.
Check that a Claude SearchBot hit is real
However, the help page names the token but does not show a full user agent string. To verify a request, check its IP address against claude.com/crawling/bots.json. Anthropic says a source IP on that list means the crawler came from Anthropic.
If the address does not match, then a third party is using the name. In that case robots.txt will not help, because the impostor has no reason to read it.
Claude SearchBot in Pagereckon
Pagereckon also files the crawler under AI search, next to OAI-SearchBot and PerplexityBot. The Assistants card shows Claude's search crawls, its live reads and the visits from claude.ai in one row.
Then watch the gap between those numbers. If the search crawler reads a page often but Claude never sends a visitor, the page may lack a short, quotable answer. Start with the AI crawler checker, then compare with the full AI bots list.
Claude SearchBot questions
Is Claude SearchBot used to train Claude?
Anthropic describes it as a search crawler. Training is the job of ClaudeBot, which has its own robots.txt token.
What happens if I block Claude SearchBot?
Anthropic says Claude cannot index your content for search, which may reduce your visibility in its search results.
Does it obey robots.txt?
Yes. Anthropic says its bots honor robots.txt directives.
Where can I see its visits?
In the free Pagereckon plugin, under AI search in the AI crawlers view. Download the free plugin to start.