AI bots that read your site, and what each one wants
This list covers the AI bots and crawlers that show up most in WordPress access logs. Each page tells you who runs the bot, why it visits, its user agent and robots.txt token, how to verify it and how to block it.
Every fact comes from the operator's own documentation, linked on the page. Also, when an operator publishes nothing, the page says so instead of guessing.
AI bots that collect training data
These crawlers read pages so their operator can train models. Blocking them keeps your text out of future training runs, but it does not remove what they already read.
| Bot | Operator | Type | robots.txt token | Follows robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | AI training crawler | GPTBot | Yes |
| ClaudeBot | Anthropic | AI training crawler | ClaudeBot | Yes |
| Google-Extended | AI training crawler | Google-Extended | Yes | |
| Applebot-Extended | Apple | AI training crawler | Applebot-Extended | Yes |
AI search bots
These build the index behind AI answers. So if you block them, the assistant may stop citing and linking your pages.
| Bot | Operator | Type | robots.txt token | Follows robots.txt |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | AI search indexer | OAI-SearchBot | Yes |
| Claude-SearchBot | Anthropic | AI search indexer | Claude-SearchBot | Yes |
| PerplexityBot | Perplexity | AI search indexer | PerplexityBot | Yes |
Live fetchers: a person asked about your page
These fetch one page at the moment a user asks the assistant about it. As a result, every request stands for a real question.
| Bot | Operator | Type | robots.txt token | Follows robots.txt |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | Live fetch for a user | ChatGPT-User | Partly |
| Claude-User | Anthropic | Live fetch for a user | Claude-User | Yes |
| Perplexity-User | Perplexity | Live fetch for a user | Perplexity-User | No |
| DuckAssistBot | DuckDuckGo | Live fetch for a user | DuckAssistBot | Yes |
| MistralAI-User | Mistral AI | Live fetch for a user | MistralAI-User | Yes |
Search engines and crawlers with several jobs
Classic search crawlers and bots whose operator lists more than one purpose, or none at all.
| Bot | Operator | Type | robots.txt token | Follows robots.txt |
|---|---|---|---|---|
| Googlebot | Search engine crawler | Googlebot | Yes | |
| Bingbot | Microsoft | Search engine crawler | bingbot | Yes |
| Bytespider | ByteDance | Several purposes | Bytespider | Not stated |
| CCBot | Common Crawl | Several purposes | CCBot | Yes |
| meta-externalagent | Meta | Several purposes | meta-externalagent | Yes |
| Amazonbot | Amazon | Several purposes | Amazonbot | Yes |
| cohere-ai | Cohere | Several purposes | cohere-ai | Not stated |
| Diffbot | Diffbot | Search engine crawler | Diffbot | Partly |
See these AI bots in your own log
The free Pagereckon plugin reads your access log and shows each crawler per page, so you see which posts GPTBot or ClaudeBot read most. In addition, the AI crawler checker explains how the three groups differ and how to spot a fake.
For blocking, start with the robots.txt rules from Google's robots.txt guide. Then use a server rule only for bots that ignore it.
Questions about AI bots
How do I see which AI bots visit my WordPress site?
Your server access log records every request with its user agent. Pagereckon reads that log and groups AI bots per page into training, AI search and live reads, so you see them without a tracking script.
Should I block every AI crawler?
Usually not. Training crawlers give little back, so many site owners block them. However, AI search bots and live fetchers are how assistants cite and link you, so blocking those can cost you visits.
Can a scraper pretend to be one of these crawlers?
Yes. A user agent is just text, so anyone can copy it. That is why each page here shows the operator's own way to verify a request, such as a published IP list or a reverse DNS check.
Does robots.txt stop AI bots?
Only bots that choose to follow it. Most operators on this list say they respect robots.txt, but some live fetchers do not, because a person asked for the page. For those, a server rule is the only hard block.
Keep reading
See your own AI traffic in ten minutes
Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.