Skip to content
AI bots / Anthropic

Claude SearchBot: how Claude finds pages to cite

Claude SearchBot is the Anthropic crawler that reads the web to improve the quality and relevance of Claude's search results. If ClaudeBot is about training, this one is about being found when someone searches through Claude.

Anthropic introduced it as a separate token, so you can treat search and training differently. Below is what that choice means for a WordPress site.

The Pagereckon team Updated
Claude searchbot at a glance, from Pagereckon
User-agent string
Claude-SearchBot
Operator
Anthropic
Type
AI search indexer
robots.txt token
Claude-SearchBot
Obeys robots.txt
Yes, by the operator's documentation
Official docs
support.claude.com

Search, not training

Anthropic gives each of its three agents one job. First, this crawler analyzes pages to make search answers more relevant and accurate. Training data comes from ClaudeBot, and live reads for one question come from Claude-User.

So the usual setup for publishers who want citations but not training looks like this:

robots.txt: no training, stay in Claude search
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Since Anthropic says its bots honor robots.txt, those few lines are all you need. There is no plugin and no server rule involved.

What blocking Claude SearchBot costs

Anthropic spells out the trade. If you disable the crawler, its system cannot index your content for search, and your site may lose visibility and accuracy in Claude's search results.

In practice, this means Claude may cite another page on your topic instead of yours. It may also describe your product from older or second-hand sources. For a site that wants to be the reference on a topic, that is a real cost.

robots.txt: leave Claude search
User-agent: Claude-SearchBot
Disallow: /

We leave out a server rule here on purpose. Anthropic warns that IP blocks stop its bots from reading robots.txt, which can undo a clean opt-out. Likewise, Pagereckon Pro's one-click blocking only covers bots that crawl hard and give nothing back, such as Bytespider.

Check that a Claude SearchBot hit is real

However, the help page names the token but does not show a full user agent string. To verify a request, check its IP address against claude.com/crawling/bots.json. Anthropic says a source IP on that list means the crawler came from Anthropic.

If the address does not match, then a third party is using the name. In that case robots.txt will not help, because the impostor has no reason to read it.

Claude SearchBot in Pagereckon

Pagereckon also files the crawler under AI search, next to OAI-SearchBot and PerplexityBot. The Assistants card shows Claude's search crawls, its live reads and the visits from claude.ai in one row.

Then watch the gap between those numbers. If the search crawler reads a page often but Claude never sends a visitor, the page may lack a short, quotable answer. Start with the AI crawler checker, then compare with the full AI bots list.

Claude SearchBot questions

Is Claude SearchBot used to train Claude?

Anthropic describes it as a search crawler. Training is the job of ClaudeBot, which has its own robots.txt token.

What happens if I block Claude SearchBot?

Anthropic says Claude cannot index your content for search, which may reduce your visibility in its search results.

Does it obey robots.txt?

Yes. Anthropic says its bots honor robots.txt directives.

Where can I see its visits?

In the free Pagereckon plugin, under AI search in the AI crawlers view. Download the free plugin to start.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.