Skip to content
AI bots / OpenAI

GPTBot: OpenAI's training crawler, explained

GPTBot is the crawler OpenAI uses to collect public web pages that may help train its foundation models. It does not fetch pages for ChatGPT answers in real time, and it does not decide whether ChatGPT search shows your site.

That split matters. OpenAI runs three separate bots, and each one has its own switch in robots.txt. So you can say no to training and still stay visible in ChatGPT search.

The Pagereckon team Updated
Gptbot at a glance, from Pagereckon
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Operator
OpenAI
Type
AI training crawler
robots.txt token
GPTBot
Obeys robots.txt
Yes, by the operator's documentation. OpenAI says a Disallow for GPTBot tells it your content should not go into training

What GPTBot does on your site

The crawler requests your pages much like a search engine would. Then it reads the HTML, follows links and comes back over time. OpenAI states that the content it gathers may be used to train its generative models.

In contrast, the other two OpenAI agents have different jobs. OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User opens a page only when a person asks ChatGPT something that needs it.

OpenAI's three main agents and what each one controls
AgentJobrobots.txt
GPTBotCollects pages for model trainingObeyed
OAI-SearchBotIndexes pages for ChatGPT searchObeyed
ChatGPT-UserFetches a page for one user requestMay not apply, per OpenAI

OpenAI also notes that when you allow both GPTBot and OAI-SearchBot, it may use one crawl for both purposes. As a result, you may see fewer visits than you expect from each name.

How to block GPTBot in WordPress

The clean way is robots.txt, because OpenAI says the crawler obeys it. Add these two lines to the robots.txt at your site root. On WordPress, an SEO plugin such as Yoast or Rank Math lets you edit that file from the dashboard.

robots.txt: opt out of OpenAI training
User-agent: GPTBot
Disallow: /

Then give it time. OpenAI says its systems can need about a day to pick up a robots.txt change. Also, the rule covers future crawls only, so it does not remove pages that were collected earlier.

A server rule as a second layer

Some site owners want a hard stop as well. A server rule answers 403 to any request whose user agent contains the token. Use the Apache or LiteSpeed version in .htaccess, or the Nginx version inside your server block.

.htaccess (Apache, LiteSpeed)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC]
RewriteRule .* - [F,L]
</IfModule>
Nginx server block
if ($http_user_agent ~* "GPTBot") {
    return 403;
}

Keep in mind that robots.txt is enough for this agent. Pagereckon Pro offers one-click blocking (an .htaccess section or an Nginx snippet) for bots that crawl hard and give nothing back, such as Bytespider. For a crawler that follows the rules, the robots.txt line above is the right tool.

What blocking GPTBot costs you

Very little in visitors, since a training crawl sends no clicks on its own. The trade is about the future: your writing will not be part of what OpenAI models learn from.

However, do not block the wrong token by accident. If you disallow OAI-SearchBot too, OpenAI says your pages will not appear in ChatGPT search answers, apart from plain navigational links. So many publishers block GPTBot and allow OAI-SearchBot on purpose.

How to verify a real GPTBot request

Anyone can copy a user agent string, so a line that says GPTBot in your log is a claim, not proof. OpenAI publishes the IP ranges its crawler uses in a JSON file at openai.com/gptbot.json.

To check a visit, take its IP address and see if it falls inside one of the listed ranges. If it does not, a scraper is borrowing the name. In that case robots.txt will not help, and a firewall or server rule is the better answer.

GPTBot in Pagereckon's AI crawler report

Pagereckon reads your server access log, so it sees every crawler request, including the ones that run no JavaScript. It files GPTBot under training, next to the AI search and live read groups for each assistant.

For example, you see which posts the training crawler reads most and how often it returns. Since training crawls send no visitors, a high count here is not a traffic signal. A live read is different: a ChatGPT-User hit means a person asked ChatGPT about that page. Try the AI crawler checker to see the full split, or browse all AI bots we track.

GPTBot questions

Does blocking GPTBot remove my site from ChatGPT?

No. ChatGPT search uses OAI-SearchBot, and live page reads use ChatGPT-User. A Disallow for the training crawler only opts you out of model training.

Does GPTBot respect robots.txt?

Yes. OpenAI says it honors a Disallow for its token, and changes can take about 24 hours to apply.

Can I block training but keep ChatGPT search?

Yes. Disallow GPTBot and leave OAI-SearchBot allowed. OpenAI treats each setting on its own.

Is Pagereckon free for tracking AI crawlers?

Yes. The AI crawler report is part of the free plugin. Download the free plugin and pick your access log to start.

Sources and references

Free plugin, optional Pro

See your own AI traffic in ten minutes

Install the free plugin, point it at your access log and read today's humans, bots and AI visits. Then add Pro when you want Search Console and the before and after reports.