Skip to content
AI crawler checker

AI crawler checker: which AI bots read which pages

An AI crawler checker tells you which AI bots visit your site and what they read. Pagereckon finds them in your WordPress server log, page by page, and splits them into training crawls, AI search indexes and live reads.

So you can see more than a yes or no. You learn which posts GPTBot or ClaudeBot read most, which pages ChatGPT opens live for its users, and which visitors only pretend to be an AI bot.

The Pagereckon team Updated
Ai crawler checker at a glance, from Pagereckon

Three kinds of AI crawlers, three different reasons

Not every AI bot wants the same thing. Some collect pages to train a model. Others build a search index so an assistant can cite you. A third group opens one page right now, because a person asked an assistant about it.

AI crawlers by purpose, as their operators describe them
OperatorTrainingAI search indexLive read for a user
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
PerplexityNone listedPerplexityBotPerplexity-User
GoogleGoogle-Extended (robots.txt token only)

This split matters when you decide what to allow. For example, OpenAI says that disallowing GPTBot signals that your content should not be used for training. Blocking OAI-SearchBot, on the other hand, keeps your pages out of ChatGPT search answers. Read each bot page for the details: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.

Google-Extended is a special case. Google says it is only a robots.txt token, with no user agent of its own, and that it does not affect Google Search. So you will never see it in a log. See the Google-Extended page for how it works.

What the AI crawler checker in Pagereckon shows

Pagereckon reads every request in your access log and matches the user agent against a long list of known bots. Then it files each AI bot under training, AI search or live read.

  • Per bot: how many requests and how many pages each AI crawler read, by day.
  • Per page: which AI crawlers read a post, so you see what interests them.
  • Live reads vs people sent: how often ChatGPT, Claude or Perplexity opened your page for a user, next to the visits those assistants sent.
  • 24-hour view: AI bots and Googlebot by hour for the last day.
  • Page panel: the admin bar on each post shows its own bot reads for 24 hours.

The free plugin includes all of that. Also, it lists other AI crawlers such as Bytespider, CCBot, Applebot-Extended, Meta-ExternalAgent and Amazonbot. The full AI crawler list explains each one.

Fake AI bots and how an AI crawler checker spots them

A user agent is just text the client sends. Anyone can write GPTBot or Googlebot into a script, and many scrapers do, because site owners rarely block well-known crawlers. As a result, a checker that trusts the user agent alone will overcount real AI bots.

The real test is the IP address. OpenAI, Perplexity and others publish JSON lists of the IP ranges their bots use. Google offers a second method: a reverse DNS lookup of the IP, which should end in a Google domain, followed by a forward lookup that returns the same IP. Our post on fake Googlebot walks through that check.

In addition, Pagereckon watches behavior. Bursts of requests, HEAD requests, outdated browsers and pages opened without CSS or images all flag a disguised bot, so it never counts as a reader.

robots.txt vs blocking at the server

robots.txt is a polite request. Reputable crawlers follow it, but the file cannot force anyone. Google also notes that it is not a way to hide a page from search results. Still, it is the opt-out most AI companies document, and Anthropic advises against blocking its IP addresses, because then its bots cannot read your robots.txt.

Live reads are different again. OpenAI says robots.txt rules may not apply to ChatGPT-User, since a person started the request. Perplexity says Perplexity-User generally ignores robots.txt for the same reason.

So use robots.txt first for well-behaved bots, and block at the server only for bots that ignore it or fake their name. The free plugin shows which bots look suspicious. One-click blocking, as an .htaccess section or an Nginx snippet, is part of Pagereckon Pro.

AI crawler checker FAQ

Is the Pagereckon AI crawler checker free?

Yes. AI crawlers per page, the training, AI search and live read split, and the suspicious bot list are all in the free plugin. One-click blocking is part of Pro.

How do I know if GPTBot really visited my site?

Compare the request IP with the ranges OpenAI publishes for GPTBot. When the user agent says GPTBot but the IP falls outside those ranges, a scraper is faking the name.

Does blocking GPTBot remove my site from ChatGPT search?

Not by itself. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training, so you can allow one and disallow the other in robots.txt.

Why can my AI crawler checker not see Google-Extended?

Google-Extended has no user agent of its own. Google crawls with its usual crawlers, and the token only tells Google how it may use the pages in Gemini.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.